2024-01-24T14:04:54.469 INFO:root:teuthology version: 0.0.1.dev251+gc856e0f 2024-01-24T14:04:54.469 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_vshankar@teuthology --verbose --archive /home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530837 --name vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi --description fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/kernel_untar_build}} -- /home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530837/orig.config.yaml 2024-01-24T14:04:54.525 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-24T14:04:54.627 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530837 branch: wip-vshankar-testing1-reef-2024-01-24-1150 description: fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/kernel_untar_build}} email: vshankar@redhat.com first_in_suite: false job_id: '7530837' kernel: client: sha1: distro flavor: default kdb: true sha1: f2d3460408951339ac0933340ad449487c9efddc ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-vshankar-testing1-reef-2024-01-24-1150 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - slow metadata IO - SLOW_OPS - slow request sha1: 4e3182a65800aed6d4aa2f6b86da3cfd72188134 subvols: create: 2 subvol_options: --namespace-isolated ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.dir_split install: ceph: flavor: default sha1: 4e3182a65800aed6d4aa2f6b86da3cfd72188134 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - nowsync syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-vshankar-testing1-reef-2024-01-24-1150 sha1: 4e3182a65800aed6d4aa2f6b86da3cfd72188134 owner: scheduled_vshankar@teuthology priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 4668 sha1: 4e3182a65800aed6d4aa2f6b86da3cfd72188134 sleep_before_teardown: 0 subset: 346/1024 suite: fs suite_branch: wip-vshankar-testing1-reef-2024-01-24-1150 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_4e3182a65800aed6d4aa2f6b86da3cfd72188134/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 4e3182a65800aed6d4aa2f6b86da3cfd72188134 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi019.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHCj2vo9mBN4fsm+73SKJZM3YxS2M2g3IWt5gbUm6jW2ldYtekXzlf1XSmHQJrb7pQ24ra9oIpY2GFVq0Gz8r6M= smithi123.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBlD4CfikN1KNTYaVpAXMZ3p6qUfCC9hNLeMCEWVmFr6xcOslMZQhCDVE1zB4pZelVZoRtaXFpPsTqfx/brODcA= smithi194.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAolgJUQEL2wZGHALo6/eF3ZJsFX6UwtYj4ZOuFPcXK8EvqTQ6Bc+sopfQrEDTmDpL1/QTlLkV7BOHONov0zftE= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:rhel8 --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - kernel_untar_build.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: c856e0f9244dd29a50697123b480d63f16a4ed81 timestamp: 2024-01-24_12:12:30 tube: smithi user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-01-24T14:04:54.628 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_4e3182a65800aed6d4aa2f6b86da3cfd72188134/qa; will attempt to use it 2024-01-24T14:04:54.628 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_4e3182a65800aed6d4aa2f6b86da3cfd72188134/qa/tasks 2024-01-24T14:04:54.629 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-24T14:04:54.630 INFO:teuthology.task.internal:Checking packages... 2024-01-24T14:04:54.653 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '4e3182a65800aed6d4aa2f6b86da3cfd72188134' 2024-01-24T14:04:54.654 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-24T14:04:54.654 INFO:teuthology.packaging:ref: None 2024-01-24T14:04:54.654 INFO:teuthology.packaging:tag: None 2024-01-24T14:04:54.654 INFO:teuthology.packaging:branch: wip-vshankar-testing1-reef-2024-01-24-1150 2024-01-24T14:04:54.654 INFO:teuthology.packaging:sha1: 4e3182a65800aed6d4aa2f6b86da3cfd72188134 2024-01-24T14:04:54.654 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-vshankar-testing1-reef-2024-01-24-1150 2024-01-24T14:04:54.795 INFO:teuthology.task.internal:Found packages for ceph version 18.2.1-341.g4e3182a6 2024-01-24T14:04:54.796 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-24T14:04:54.803 INFO:teuthology.task.internal:no buildpackages task found 2024-01-24T14:04:54.803 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-24T14:04:54.856 INFO:teuthology.task.internal:Saving configuration 2024-01-24T14:04:54.876 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-24T14:04:54.883 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-24T14:04:54.908 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi019.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530837', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-24 13:59:42.252661', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHCj2vo9mBN4fsm+73SKJZM3YxS2M2g3IWt5gbUm6jW2ldYtekXzlf1XSmHQJrb7pQ24ra9oIpY2GFVq0Gz8r6M='} 2024-01-24T14:04:54.934 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi123.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530837', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-24 13:59:42.254693', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBlD4CfikN1KNTYaVpAXMZ3p6qUfCC9hNLeMCEWVmFr6xcOslMZQhCDVE1zB4pZelVZoRtaXFpPsTqfx/brODcA='} 2024-01-24T14:04:54.959 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi194.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530837', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-24 13:59:42.256410', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAolgJUQEL2wZGHALo6/eF3ZJsFX6UwtYj4ZOuFPcXK8EvqTQ6Bc+sopfQrEDTmDpL1/QTlLkV7BOHONov0zftE='} 2024-01-24T14:04:54.959 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-24T14:04:54.966 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-01-24T14:04:54.967 INFO:teuthology.task.internal:roles: ubuntu@smithi123.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-01-24T14:04:54.967 INFO:teuthology.task.internal:roles: ubuntu@smithi194.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-01-24T14:04:54.967 INFO:teuthology.run_tasks:Running task console_log... 2024-01-24T14:04:55.084 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4cf8f8ef70>, signals=[15]) 2024-01-24T14:04:55.085 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-24T14:04:55.092 INFO:teuthology.task.internal:Opening connections... 2024-01-24T14:04:55.093 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi019.front.sepia.ceph.com 2024-01-24T14:04:55.094 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-24T14:04:55.169 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi123.front.sepia.ceph.com 2024-01-24T14:04:55.170 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-24T14:04:55.243 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi194.front.sepia.ceph.com 2024-01-24T14:04:55.244 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-24T14:04:55.318 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-24T14:04:55.327 DEBUG:teuthology.orchestra.run.smithi019:> uname -m 2024-01-24T14:04:55.348 INFO:teuthology.orchestra.run.smithi019.stdout:x86_64 2024-01-24T14:04:55.348 DEBUG:teuthology.orchestra.run.smithi019:> cat /etc/os-release 2024-01-24T14:04:55.409 INFO:teuthology.orchestra.run.smithi019.stdout:NAME="Red Hat Enterprise Linux" 2024-01-24T14:04:55.409 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION="8.6 (Ootpa)" 2024-01-24T14:04:55.410 INFO:teuthology.orchestra.run.smithi019.stdout:ID="rhel" 2024-01-24T14:04:55.410 INFO:teuthology.orchestra.run.smithi019.stdout:ID_LIKE="fedora" 2024-01-24T14:04:55.410 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION_ID="8.6" 2024-01-24T14:04:55.410 INFO:teuthology.orchestra.run.smithi019.stdout:PLATFORM_ID="platform:el8" 2024-01-24T14:04:55.410 INFO:teuthology.orchestra.run.smithi019.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-01-24T14:04:55.410 INFO:teuthology.orchestra.run.smithi019.stdout:ANSI_COLOR="0;31" 2024-01-24T14:04:55.410 INFO:teuthology.orchestra.run.smithi019.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-01-24T14:04:55.410 INFO:teuthology.orchestra.run.smithi019.stdout:HOME_URL="https://www.redhat.com/" 2024-01-24T14:04:55.410 INFO:teuthology.orchestra.run.smithi019.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-01-24T14:04:55.410 INFO:teuthology.orchestra.run.smithi019.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-24T14:04:55.410 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:04:55.410 INFO:teuthology.orchestra.run.smithi019.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-24T14:04:55.410 INFO:teuthology.orchestra.run.smithi019.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-01-24T14:04:55.410 INFO:teuthology.orchestra.run.smithi019.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-01-24T14:04:55.411 INFO:teuthology.orchestra.run.smithi019.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-01-24T14:04:55.411 INFO:teuthology.lock.ops:Updating smithi019.front.sepia.ceph.com on lock server 2024-01-24T14:04:55.439 DEBUG:teuthology.orchestra.run.smithi123:> uname -m 2024-01-24T14:04:55.460 INFO:teuthology.orchestra.run.smithi123.stdout:x86_64 2024-01-24T14:04:55.461 DEBUG:teuthology.orchestra.run.smithi123:> cat /etc/os-release 2024-01-24T14:04:55.521 INFO:teuthology.orchestra.run.smithi123.stdout:NAME="Red Hat Enterprise Linux" 2024-01-24T14:04:55.521 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION="8.6 (Ootpa)" 2024-01-24T14:04:55.521 INFO:teuthology.orchestra.run.smithi123.stdout:ID="rhel" 2024-01-24T14:04:55.521 INFO:teuthology.orchestra.run.smithi123.stdout:ID_LIKE="fedora" 2024-01-24T14:04:55.521 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION_ID="8.6" 2024-01-24T14:04:55.521 INFO:teuthology.orchestra.run.smithi123.stdout:PLATFORM_ID="platform:el8" 2024-01-24T14:04:55.521 INFO:teuthology.orchestra.run.smithi123.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-01-24T14:04:55.522 INFO:teuthology.orchestra.run.smithi123.stdout:ANSI_COLOR="0;31" 2024-01-24T14:04:55.522 INFO:teuthology.orchestra.run.smithi123.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-01-24T14:04:55.522 INFO:teuthology.orchestra.run.smithi123.stdout:HOME_URL="https://www.redhat.com/" 2024-01-24T14:04:55.522 INFO:teuthology.orchestra.run.smithi123.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-01-24T14:04:55.522 INFO:teuthology.orchestra.run.smithi123.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-24T14:04:55.522 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-24T14:04:55.522 INFO:teuthology.orchestra.run.smithi123.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-24T14:04:55.522 INFO:teuthology.orchestra.run.smithi123.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-01-24T14:04:55.522 INFO:teuthology.orchestra.run.smithi123.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-01-24T14:04:55.522 INFO:teuthology.orchestra.run.smithi123.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-01-24T14:04:55.522 INFO:teuthology.lock.ops:Updating smithi123.front.sepia.ceph.com on lock server 2024-01-24T14:04:55.555 DEBUG:teuthology.orchestra.run.smithi194:> uname -m 2024-01-24T14:04:55.576 INFO:teuthology.orchestra.run.smithi194.stdout:x86_64 2024-01-24T14:04:55.576 DEBUG:teuthology.orchestra.run.smithi194:> cat /etc/os-release 2024-01-24T14:04:55.636 INFO:teuthology.orchestra.run.smithi194.stdout:NAME="Red Hat Enterprise Linux" 2024-01-24T14:04:55.636 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION="8.6 (Ootpa)" 2024-01-24T14:04:55.636 INFO:teuthology.orchestra.run.smithi194.stdout:ID="rhel" 2024-01-24T14:04:55.636 INFO:teuthology.orchestra.run.smithi194.stdout:ID_LIKE="fedora" 2024-01-24T14:04:55.637 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION_ID="8.6" 2024-01-24T14:04:55.637 INFO:teuthology.orchestra.run.smithi194.stdout:PLATFORM_ID="platform:el8" 2024-01-24T14:04:55.637 INFO:teuthology.orchestra.run.smithi194.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-01-24T14:04:55.637 INFO:teuthology.orchestra.run.smithi194.stdout:ANSI_COLOR="0;31" 2024-01-24T14:04:55.637 INFO:teuthology.orchestra.run.smithi194.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-01-24T14:04:55.637 INFO:teuthology.orchestra.run.smithi194.stdout:HOME_URL="https://www.redhat.com/" 2024-01-24T14:04:55.637 INFO:teuthology.orchestra.run.smithi194.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-01-24T14:04:55.637 INFO:teuthology.orchestra.run.smithi194.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-24T14:04:55.637 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-24T14:04:55.637 INFO:teuthology.orchestra.run.smithi194.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-24T14:04:55.637 INFO:teuthology.orchestra.run.smithi194.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-01-24T14:04:55.637 INFO:teuthology.orchestra.run.smithi194.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-01-24T14:04:55.637 INFO:teuthology.orchestra.run.smithi194.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-01-24T14:04:55.638 INFO:teuthology.lock.ops:Updating smithi194.front.sepia.ceph.com on lock server 2024-01-24T14:04:55.655 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-24T14:04:55.667 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-24T14:04:55.689 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-24T14:04:55.689 DEBUG:teuthology.orchestra.run.smithi019:> test '!' -e /home/ubuntu/cephtest 2024-01-24T14:04:55.692 DEBUG:teuthology.orchestra.run.smithi123:> test '!' -e /home/ubuntu/cephtest 2024-01-24T14:04:55.694 DEBUG:teuthology.orchestra.run.smithi194:> test '!' -e /home/ubuntu/cephtest 2024-01-24T14:04:55.713 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-24T14:04:55.720 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-24T14:04:55.721 DEBUG:teuthology.orchestra.run.smithi019:> test -z $(ls -A /var/lib/ceph) 2024-01-24T14:04:55.753 DEBUG:teuthology.orchestra.run.smithi123:> test -z $(ls -A /var/lib/ceph) 2024-01-24T14:04:55.755 DEBUG:teuthology.orchestra.run.smithi194:> test -z $(ls -A /var/lib/ceph) 2024-01-24T14:04:55.782 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-24T14:04:55.866 INFO:teuthology.run_tasks:Running task kernel... 2024-01-24T14:04:55.885 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': 'f2d3460408951339ac0933340ad449487c9efddc'} 2024-01-24T14:04:55.885 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-01-24T14:04:55.885 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-01-24T14:04:55.885 DEBUG:teuthology.orchestra.run.smithi019:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-24T14:04:55.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T14:04:55.906 DEBUG:teuthology.orchestra.run.smithi019:> uname -r 2024-01-24T14:04:55.967 INFO:teuthology.orchestra.run.smithi019.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-01-24T14:04:55.967 INFO:teuthology.task.kernel:Running kernel on smithi019: 4.18.0-513.9.1.el8_9.x86_64 2024-01-24T14:04:55.967 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum install -y kernel 2024-01-24T14:04:57.581 INFO:teuthology.orchestra.run.smithi019.stdout:Updating Subscription Management repositories. 2024-01-24T14:04:57.581 INFO:teuthology.orchestra.run.smithi019.stdout:Unable to read consumer identity 2024-01-24T14:04:57.894 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:04:57.894 INFO:teuthology.orchestra.run.smithi019.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-24T14:04:57.894 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:04:59.699 INFO:teuthology.orchestra.run.smithi019.stdout:Copr repo for python3-asyncssh owned by ceph 19 kB/s | 3.5 kB 00:00 2024-01-24T14:05:00.508 INFO:teuthology.orchestra.run.smithi019.stdout:Extra Packages for Enterprise Linux 21 MB/s | 16 MB 00:00 2024-01-24T14:05:04.441 INFO:teuthology.orchestra.run.smithi019.stdout:lab-extras 444 kB/s | 24 kB 00:00 2024-01-24T14:05:05.319 INFO:teuthology.orchestra.run.smithi019.stdout:Package kernel-4.18.0-513.9.1.el8_9.x86_64 is already installed. 2024-01-24T14:05:05.319 INFO:teuthology.orchestra.run.smithi019.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2024-01-24T14:05:05.397 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-01-24T14:05:05.399 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-01-24T14:05:05.399 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-01-24T14:05:05.541 DEBUG:teuthology.orchestra.run.smithi019:> echo no | sudo yum reinstall kernel || true 2024-01-24T14:05:05.928 INFO:teuthology.orchestra.run.smithi019.stdout:Updating Subscription Management repositories. 2024-01-24T14:05:05.929 INFO:teuthology.orchestra.run.smithi019.stdout:Unable to read consumer identity 2024-01-24T14:05:06.180 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:05:06.180 INFO:teuthology.orchestra.run.smithi019.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-24T14:05:06.180 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:05:06.548 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:00:02 ago on Wed 24 Jan 2024 02:05:04 PM UTC. 2024-01-24T14:05:06.625 INFO:teuthology.orchestra.run.smithi019.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-01-24T14:05:06.641 INFO:teuthology.orchestra.run.smithi019.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-01-24T14:05:06.682 INFO:teuthology.orchestra.run.smithi019.stderr:Error: No packages marked for reinstall. 2024-01-24T14:05:06.738 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum reinstall -y kernel || true 2024-01-24T14:05:07.120 INFO:teuthology.orchestra.run.smithi019.stdout:Updating Subscription Management repositories. 2024-01-24T14:05:07.120 INFO:teuthology.orchestra.run.smithi019.stdout:Unable to read consumer identity 2024-01-24T14:05:07.206 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:05:07.206 INFO:teuthology.orchestra.run.smithi019.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-01-24T14:05:07.206 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:05:07.559 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:00:03 ago on Wed 24 Jan 2024 02:05:04 PM UTC. 2024-01-24T14:05:07.628 INFO:teuthology.orchestra.run.smithi019.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-01-24T14:05:07.628 INFO:teuthology.orchestra.run.smithi019.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-01-24T14:05:07.662 INFO:teuthology.orchestra.run.smithi019.stderr:Error: No packages marked for reinstall. 2024-01-24T14:05:07.717 DEBUG:teuthology.orchestra.run.smithi019:> rpm -q kernel | sort -rV | head -n 1 2024-01-24T14:05:07.889 INFO:teuthology.orchestra.run.smithi019.stdout:kernel-4.18.0-513.9.1.el8_9.x86_64 2024-01-24T14:05:07.889 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.9.1.el8_9.x86_64 2024-01-24T14:05:07.890 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-01-24T14:05:07.890 DEBUG:teuthology.parallel:result is None 2024-01-24T14:05:07.890 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-24T14:05:07.898 INFO:teuthology.task.internal:Creating test directory... 2024-01-24T14:05:07.898 DEBUG:teuthology.orchestra.run.smithi019:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-24T14:05:07.900 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-24T14:05:07.903 DEBUG:teuthology.orchestra.run.smithi194:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-24T14:05:07.923 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-24T14:05:07.932 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-24T14:05:07.960 INFO:teuthology.task.internal:Creating archive directory... 2024-01-24T14:05:07.961 DEBUG:teuthology.orchestra.run.smithi019:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-24T14:05:07.963 DEBUG:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-24T14:05:07.965 DEBUG:teuthology.orchestra.run.smithi194:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-24T14:05:07.998 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-24T14:05:08.007 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-24T14:05:08.007 DEBUG:teuthology.orchestra.run.smithi123:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-24T14:05:08.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T14:05:08.051 DEBUG:teuthology.orchestra.run.smithi194:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-24T14:05:08.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T14:05:08.070 DEBUG:teuthology.orchestra.run.smithi019:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-24T14:05:08.072 DEBUG:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-24T14:05:08.093 DEBUG:teuthology.orchestra.run.smithi194:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-24T14:05:08.105 INFO:teuthology.orchestra.run.smithi019.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-24T14:05:08.123 INFO:teuthology.orchestra.run.smithi019.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-24T14:05:08.137 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-24T14:05:08.150 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-24T14:05:08.171 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-24T14:05:08.183 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-24T14:05:08.185 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-24T14:05:08.192 INFO:teuthology.task.internal:Configuring sudo... 2024-01-24T14:05:08.192 DEBUG:teuthology.orchestra.run.smithi019:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-24T14:05:08.195 DEBUG:teuthology.orchestra.run.smithi123:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-24T14:05:08.197 DEBUG:teuthology.orchestra.run.smithi194:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-24T14:05:08.241 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-24T14:05:08.251 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-24T14:05:08.251 DEBUG:teuthology.orchestra.run.smithi019:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-24T14:05:08.270 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-24T14:05:08.272 DEBUG:teuthology.orchestra.run.smithi194:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-24T14:05:08.300 DEBUG:teuthology.orchestra.run.smithi019:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-24T14:05:08.376 DEBUG:teuthology.orchestra.run.smithi019:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-24T14:05:08.439 DEBUG:teuthology.orchestra.run.smithi019:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-24T14:05:08.483 DEBUG:teuthology.orchestra.run.smithi019:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-24T14:05:08.538 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-01-24T14:05:08.538 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-24T14:05:08.612 DEBUG:teuthology.orchestra.run.smithi123:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-24T14:05:08.658 DEBUG:teuthology.orchestra.run.smithi123:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-24T14:05:08.715 DEBUG:teuthology.orchestra.run.smithi123:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-24T14:05:08.797 DEBUG:teuthology.orchestra.run.smithi123:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-24T14:05:08.842 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-01-24T14:05:08.842 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-24T14:05:08.909 DEBUG:teuthology.orchestra.run.smithi194:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-24T14:05:08.957 DEBUG:teuthology.orchestra.run.smithi194:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-24T14:05:09.006 DEBUG:teuthology.orchestra.run.smithi194:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-24T14:05:09.091 DEBUG:teuthology.orchestra.run.smithi194:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-24T14:05:09.136 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-01-24T14:05:09.136 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-24T14:05:09.205 DEBUG:teuthology.orchestra.run.smithi019:> sudo service rsyslog restart 2024-01-24T14:05:09.208 DEBUG:teuthology.orchestra.run.smithi123:> sudo service rsyslog restart 2024-01-24T14:05:09.210 DEBUG:teuthology.orchestra.run.smithi194:> sudo service rsyslog restart 2024-01-24T14:05:09.259 INFO:teuthology.orchestra.run.smithi019.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-24T14:05:09.268 INFO:teuthology.orchestra.run.smithi123.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-24T14:05:09.320 INFO:teuthology.orchestra.run.smithi194.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-24T14:05:09.751 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-24T14:05:09.759 INFO:teuthology.task.internal:Starting timer... 2024-01-24T14:05:09.759 INFO:teuthology.run_tasks:Running task pcp... 2024-01-24T14:05:09.815 INFO:teuthology.run_tasks:Running task selinux... 2024-01-24T14:05:09.848 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:logrotate_t:s0']} 2024-01-24T14:05:09.848 DEBUG:teuthology.orchestra.run.smithi019:> sudo service auditd rotate 2024-01-24T14:05:09.929 INFO:teuthology.orchestra.run.smithi019.stdout:Rotating logs: 2024-01-24T14:05:09.931 DEBUG:teuthology.orchestra.run.smithi123:> sudo service auditd rotate 2024-01-24T14:05:10.003 INFO:teuthology.orchestra.run.smithi123.stdout:Rotating logs: 2024-01-24T14:05:10.005 DEBUG:teuthology.orchestra.run.smithi194:> sudo service auditd rotate 2024-01-24T14:05:10.088 INFO:teuthology.orchestra.run.smithi194.stdout:Rotating logs: 2024-01-24T14:05:10.089 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-24T14:05:10.089 DEBUG:teuthology.orchestra.run.smithi019:> /usr/sbin/getenforce 2024-01-24T14:05:10.121 INFO:teuthology.orchestra.run.smithi019.stdout:Permissive 2024-01-24T14:05:10.121 DEBUG:teuthology.orchestra.run.smithi123:> /usr/sbin/getenforce 2024-01-24T14:05:10.160 INFO:teuthology.orchestra.run.smithi123.stdout:Permissive 2024-01-24T14:05:10.160 DEBUG:teuthology.orchestra.run.smithi194:> /usr/sbin/getenforce 2024-01-24T14:05:10.187 INFO:teuthology.orchestra.run.smithi194.stdout:Permissive 2024-01-24T14:05:10.187 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi019.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi123.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi194.front.sepia.ceph.com': 'permissive'} 2024-01-24T14:05:10.188 DEBUG:teuthology.orchestra.run.smithi019:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-24T14:05:10.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T14:05:10.220 DEBUG:teuthology.orchestra.run.smithi123:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-24T14:05:10.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T14:05:10.247 DEBUG:teuthology.orchestra.run.smithi194:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-24T14:05:10.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T14:05:10.276 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-24T14:05:10.276 DEBUG:teuthology.orchestra.run.smithi019:> sudo /usr/sbin/setenforce permissive 2024-01-24T14:05:10.306 DEBUG:teuthology.orchestra.run.smithi123:> sudo /usr/sbin/setenforce permissive 2024-01-24T14:05:10.345 DEBUG:teuthology.orchestra.run.smithi194:> sudo /usr/sbin/setenforce permissive 2024-01-24T14:05:10.371 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-24T14:05:10.381 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-24T14:05:10.397 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-24T14:05:10.398 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi019.front.sepia.ceph.com,smithi123.front.sepia.ceph.com,smithi194.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-24T14:14:27.572 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi019.front.sepia.ceph.com'), Remote(name='ubuntu@smithi123.front.sepia.ceph.com'), Remote(name='ubuntu@smithi194.front.sepia.ceph.com')] 2024-01-24T14:14:27.574 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-01-24T14:14:27.575 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-24T14:14:27.658 DEBUG:teuthology.orchestra.run.smithi019:> true 2024-01-24T14:14:27.734 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-01-24T14:14:27.735 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-01-24T14:14:27.735 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-24T14:14:27.808 DEBUG:teuthology.orchestra.run.smithi123:> true 2024-01-24T14:14:27.872 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-01-24T14:14:27.873 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-01-24T14:14:27.873 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-24T14:14:27.952 DEBUG:teuthology.orchestra.run.smithi194:> true 2024-01-24T14:14:28.022 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-01-24T14:14:28.023 INFO:teuthology.run_tasks:Running task clock... 2024-01-24T14:14:28.043 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-24T14:14:28.043 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-24T14:14:28.044 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-24T14:14:28.046 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-24T14:14:28.046 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-24T14:14:28.048 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-24T14:14:28.048 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-24T14:14:28.078 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-24T14:14:28.084 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-24T14:14:28.093 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-24T14:14:28.094 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-24T14:14:28.102 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-24T14:14:28.109 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-24T14:14:28.165 INFO:teuthology.orchestra.run.smithi123.stderr:sudo: ntpd: command not found 2024-01-24T14:14:28.177 INFO:teuthology.orchestra.run.smithi123.stdout:506 Cannot talk to daemon 2024-01-24T14:14:28.179 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: ntpd: command not found 2024-01-24T14:14:28.182 INFO:teuthology.orchestra.run.smithi194.stderr:sudo: ntpd: command not found 2024-01-24T14:14:28.191 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-24T14:14:28.194 INFO:teuthology.orchestra.run.smithi019.stdout:506 Cannot talk to daemon 2024-01-24T14:14:28.194 INFO:teuthology.orchestra.run.smithi194.stdout:506 Cannot talk to daemon 2024-01-24T14:14:28.205 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-24T14:14:28.209 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-24T14:14:28.212 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-24T14:14:28.223 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-24T14:14:28.229 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-24T14:14:28.293 INFO:teuthology.orchestra.run.smithi123.stderr:bash: ntpq: command not found 2024-01-24T14:14:28.295 INFO:teuthology.orchestra.run.smithi194.stderr:bash: ntpq: command not found 2024-01-24T14:14:28.296 INFO:teuthology.orchestra.run.smithi123.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-24T14:14:28.296 INFO:teuthology.orchestra.run.smithi123.stdout:=============================================================================== 2024-01-24T14:14:28.297 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-24T14:14:28.297 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-24T14:14:28.297 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-24T14:14:28.297 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-24T14:14:28.299 INFO:teuthology.orchestra.run.smithi194.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-24T14:14:28.299 INFO:teuthology.orchestra.run.smithi194.stdout:=============================================================================== 2024-01-24T14:14:28.299 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-24T14:14:28.299 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-24T14:14:28.299 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-24T14:14:28.299 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-24T14:14:28.310 INFO:teuthology.orchestra.run.smithi019.stderr:bash: ntpq: command not found 2024-01-24T14:14:28.314 INFO:teuthology.orchestra.run.smithi019.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-24T14:14:28.315 INFO:teuthology.orchestra.run.smithi019.stdout:=============================================================================== 2024-01-24T14:14:28.315 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-24T14:14:28.315 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-24T14:14:28.315 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-24T14:14:28.315 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-24T14:14:28.316 INFO:teuthology.run_tasks:Running task pexec... 2024-01-24T14:14:28.326 INFO:teuthology.task.pexec:Executing custom commands... 2024-01-24T14:14:28.326 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi019.front.sepia.ceph.com 2024-01-24T14:14:28.326 DEBUG:teuthology.orchestra.run.smithi019:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-24T14:14:28.327 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi123.front.sepia.ceph.com 2024-01-24T14:14:28.327 DEBUG:teuthology.orchestra.run.smithi123:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-24T14:14:28.327 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi194.front.sepia.ceph.com 2024-01-24T14:14:28.327 DEBUG:teuthology.orchestra.run.smithi194:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-24T14:14:28.713 INFO:teuthology.orchestra.run.smithi123.stdout:Updating Subscription Management repositories. 2024-01-24T14:14:28.719 INFO:teuthology.orchestra.run.smithi194.stdout:Updating Subscription Management repositories. 2024-01-24T14:14:28.793 INFO:teuthology.orchestra.run.smithi019.stdout:Updating Subscription Management repositories. 2024-01-24T14:14:30.634 INFO:teuthology.orchestra.run.smithi194.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 35 kB/s | 4.5 kB 00:00 2024-01-24T14:14:30.683 INFO:teuthology.orchestra.run.smithi123.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 34 kB/s | 4.1 kB 00:00 2024-01-24T14:14:30.813 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 34 kB/s | 4.5 kB 00:00 2024-01-24T14:14:31.042 INFO:teuthology.orchestra.run.smithi123.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 33 kB/s | 4.5 kB 00:00 2024-01-24T14:14:31.262 INFO:teuthology.orchestra.run.smithi194.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 38 kB/s | 4.1 kB 00:00 2024-01-24T14:14:31.461 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 34 kB/s | 4.1 kB 00:00 2024-01-24T14:14:34.088 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-24T14:14:34.089 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-24T14:14:34.089 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-01-24T14:14:34.089 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-24T14:14:34.089 INFO:teuthology.orchestra.run.smithi194.stdout:Resetting modules: 2024-01-24T14:14:34.089 INFO:teuthology.orchestra.run.smithi194.stdout: container-tools 2024-01-24T14:14:34.089 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-24T14:14:34.089 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-01-24T14:14:34.089 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-24T14:14:34.089 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-24T14:14:34.317 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-01-24T14:14:34.318 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-24T14:14:34.318 INFO:teuthology.orchestra.run.smithi123.stdout: Package Architecture Version Repository Size 2024-01-24T14:14:34.318 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-24T14:14:34.318 INFO:teuthology.orchestra.run.smithi123.stdout:Resetting modules: 2024-01-24T14:14:34.319 INFO:teuthology.orchestra.run.smithi123.stdout: container-tools 2024-01-24T14:14:34.319 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-24T14:14:34.319 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-01-24T14:14:34.319 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-01-24T14:14:34.319 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-24T14:14:34.693 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-01-24T14:14:34.695 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-01-24T14:14:34.695 INFO:teuthology.orchestra.run.smithi019.stdout: Package Architecture Version Repository Size 2024-01-24T14:14:34.695 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-01-24T14:14:34.695 INFO:teuthology.orchestra.run.smithi019.stdout:Resetting modules: 2024-01-24T14:14:34.695 INFO:teuthology.orchestra.run.smithi019.stdout: container-tools 2024-01-24T14:14:34.695 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:14:34.695 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-01-24T14:14:34.695 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-01-24T14:14:34.695 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:14:42.343 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-24T14:14:42.843 INFO:teuthology.orchestra.run.smithi194.stdout:Updating Subscription Management repositories. 2024-01-24T14:14:44.742 INFO:teuthology.orchestra.run.smithi194.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 28 kB/s | 4.5 kB 00:00 2024-01-24T14:14:45.388 INFO:teuthology.orchestra.run.smithi194.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 35 kB/s | 4.1 kB 00:00 2024-01-24T14:14:45.917 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-01-24T14:14:46.391 INFO:teuthology.orchestra.run.smithi123.stdout:Updating Subscription Management repositories. 2024-01-24T14:14:46.747 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-01-24T14:14:47.275 INFO:teuthology.orchestra.run.smithi019.stdout:Updating Subscription Management repositories. 2024-01-24T14:14:48.111 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-24T14:14:48.269 INFO:teuthology.orchestra.run.smithi194.stdout:========================================================================================================== 2024-01-24T14:14:48.270 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-01-24T14:14:48.270 INFO:teuthology.orchestra.run.smithi194.stdout:========================================================================================================== 2024-01-24T14:14:48.270 INFO:teuthology.orchestra.run.smithi194.stdout:Installing group/module packages: 2024-01-24T14:14:48.270 INFO:teuthology.orchestra.run.smithi194.stdout: aardvark-dns x86_64 2:1.7.0-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 1.0 M 2024-01-24T14:14:48.270 INFO:teuthology.orchestra.run.smithi194.stdout: buildah x86_64 1:1.31.3-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 8.8 M 2024-01-24T14:14:48.270 INFO:teuthology.orchestra.run.smithi194.stdout: cockpit-podman noarch 75-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 739 k 2024-01-24T14:14:48.270 INFO:teuthology.orchestra.run.smithi194.stdout: crun x86_64 1.8.7-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 239 k 2024-01-24T14:14:48.270 INFO:teuthology.orchestra.run.smithi194.stdout: netavark x86_64 2:1.7.0-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-01-24T14:14:48.270 INFO:teuthology.orchestra.run.smithi194.stdout: python3-podman noarch 4.6.0-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 153 k 2024-01-24T14:14:48.270 INFO:teuthology.orchestra.run.smithi194.stdout: skopeo x86_64 2:1.13.3-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 8.1 M 2024-01-24T14:14:48.270 INFO:teuthology.orchestra.run.smithi194.stdout: toolbox x86_64 0.0.99.4-5.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 2.5 M 2024-01-24T14:14:48.270 INFO:teuthology.orchestra.run.smithi194.stdout: udica noarch 0.2.6-20.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 49 k 2024-01-24T14:14:48.270 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-01-24T14:14:48.271 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2024-01-24T14:14:48.271 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2024-01-24T14:14:48.271 INFO:teuthology.orchestra.run.smithi194.stdout: yajl x86_64 2.1.0-12.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2024-01-24T14:14:48.271 INFO:teuthology.orchestra.run.smithi194.stdout:Installing weak dependencies: 2024-01-24T14:14:48.271 INFO:teuthology.orchestra.run.smithi194.stdout: criu-libs x86_64 3.18-4.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 39 k 2024-01-24T14:14:48.271 INFO:teuthology.orchestra.run.smithi194.stdout:Installing module profiles: 2024-01-24T14:14:48.271 INFO:teuthology.orchestra.run.smithi194.stdout: container-tools/common 2024-01-24T14:14:48.271 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-24T14:14:48.271 INFO:teuthology.orchestra.run.smithi194.stdout:Enabling module streams: 2024-01-24T14:14:48.271 INFO:teuthology.orchestra.run.smithi194.stdout: container-tools rhel8 2024-01-24T14:14:48.271 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-24T14:14:48.271 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-01-24T14:14:48.271 INFO:teuthology.orchestra.run.smithi194.stdout:========================================================================================================== 2024-01-24T14:14:48.271 INFO:teuthology.orchestra.run.smithi194.stdout:Install 13 Packages 2024-01-24T14:14:48.271 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-24T14:14:48.272 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 26 M 2024-01-24T14:14:48.272 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 87 M 2024-01-24T14:14:48.272 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-01-24T14:14:48.411 INFO:teuthology.orchestra.run.smithi123.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 25 kB/s | 4.1 kB 00:00 2024-01-24T14:14:48.438 INFO:teuthology.orchestra.run.smithi194.stdout:(1/13): python3-pytoml-0.1.14-5.git7dea353.el8. 150 kB/s | 25 kB 00:00 2024-01-24T14:14:48.463 INFO:teuthology.orchestra.run.smithi194.stdout:(2/13): yajl-2.1.0-12.el8.x86_64.rpm 215 kB/s | 41 kB 00:00 2024-01-24T14:14:48.489 INFO:teuthology.orchestra.run.smithi194.stdout:(3/13): python3-pyxdg-0.25-16.el8.noarch.rpm 434 kB/s | 94 kB 00:00 2024-01-24T14:14:48.606 INFO:teuthology.orchestra.run.smithi194.stdout:(4/13): aardvark-dns-1.7.0-1.module+el8.9.0+203 6.0 MB/s | 1.0 MB 00:00 2024-01-24T14:14:48.664 INFO:teuthology.orchestra.run.smithi194.stdout:(5/13): cockpit-podman-75-1.module+el8.9.0+2032 4.1 MB/s | 739 kB 00:00 2024-01-24T14:14:48.748 INFO:teuthology.orchestra.run.smithi194.stdout:(6/13): criu-libs-3.18-4.module+el8.9.0+20326+3 277 kB/s | 39 kB 00:00 2024-01-24T14:14:48.758 INFO:teuthology.orchestra.run.smithi123.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 38 kB/s | 4.5 kB 00:00 2024-01-24T14:14:48.857 INFO:teuthology.orchestra.run.smithi194.stdout:(7/13): buildah-1.31.3-1.module+el8.9.0+20326+3 22 MB/s | 8.8 MB 00:00 2024-01-24T14:14:48.882 INFO:teuthology.orchestra.run.smithi194.stdout:(8/13): crun-1.8.7-1.module+el8.9.0+20326+38708 1.1 MB/s | 239 kB 00:00 2024-01-24T14:14:48.965 INFO:teuthology.orchestra.run.smithi194.stdout:(9/13): netavark-1.7.0-1.module+el8.9.0+20326+3 17 MB/s | 3.7 MB 00:00 2024-01-24T14:14:49.032 INFO:teuthology.orchestra.run.smithi194.stdout:(10/13): python3-podman-4.6.0-1.module+el8.9.0+ 876 kB/s | 153 kB 00:00 2024-01-24T14:14:49.134 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 31 kB/s | 4.5 kB 00:00 2024-01-24T14:14:49.149 INFO:teuthology.orchestra.run.smithi194.stdout:(11/13): skopeo-1.13.3-1.module+el8.9.0+20326+3 30 MB/s | 8.1 MB 00:00 2024-01-24T14:14:49.174 INFO:teuthology.orchestra.run.smithi194.stdout:(12/13): udica-0.2.6-20.module+el8.9.0+20326+38 349 kB/s | 49 kB 00:00 2024-01-24T14:14:49.249 INFO:teuthology.orchestra.run.smithi194.stdout:(13/13): toolbox-0.0.99.4-5.module+el8.9.0+2032 8.9 MB/s | 2.5 MB 00:00 2024-01-24T14:14:49.250 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-01-24T14:14:49.250 INFO:teuthology.orchestra.run.smithi194.stdout:Total 26 MB/s | 26 MB 00:00 2024-01-24T14:14:49.536 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-01-24T14:14:49.556 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-01-24T14:14:49.557 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-01-24T14:14:49.740 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 61 kB/s | 4.1 kB 00:00 2024-01-24T14:14:49.786 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-01-24T14:14:49.795 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-01-24T14:14:50.422 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-01-24T14:14:50.691 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 1/13 2024-01-24T14:14:51.212 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : aardvark-dns-2:1.7.0-1.module+el8.9.0+20326+387084 2/13 2024-01-24T14:14:51.401 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : netavark-2:1.7.0-1.module+el8.9.0+20326+387084d0.x 3/13 2024-01-24T14:14:51.584 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : yajl-2.1.0-12.el8.x86_64 4/13 2024-01-24T14:14:51.780 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 5/13 2024-01-24T14:14:51.801 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-01-24T14:14:51.955 INFO:teuthology.orchestra.run.smithi123.stdout:========================================================================================================== 2024-01-24T14:14:51.955 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2024-01-24T14:14:51.955 INFO:teuthology.orchestra.run.smithi123.stdout:========================================================================================================== 2024-01-24T14:14:51.955 INFO:teuthology.orchestra.run.smithi123.stdout:Installing group/module packages: 2024-01-24T14:14:51.956 INFO:teuthology.orchestra.run.smithi123.stdout: aardvark-dns x86_64 2:1.7.0-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 1.0 M 2024-01-24T14:14:51.956 INFO:teuthology.orchestra.run.smithi123.stdout: buildah x86_64 1:1.31.3-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 8.8 M 2024-01-24T14:14:51.956 INFO:teuthology.orchestra.run.smithi123.stdout: cockpit-podman noarch 75-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 739 k 2024-01-24T14:14:51.956 INFO:teuthology.orchestra.run.smithi123.stdout: crun x86_64 1.8.7-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 239 k 2024-01-24T14:14:51.956 INFO:teuthology.orchestra.run.smithi123.stdout: netavark x86_64 2:1.7.0-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-01-24T14:14:51.956 INFO:teuthology.orchestra.run.smithi123.stdout: python3-podman noarch 4.6.0-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 153 k 2024-01-24T14:14:51.956 INFO:teuthology.orchestra.run.smithi123.stdout: skopeo x86_64 2:1.13.3-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 8.1 M 2024-01-24T14:14:51.956 INFO:teuthology.orchestra.run.smithi123.stdout: toolbox x86_64 0.0.99.4-5.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 2.5 M 2024-01-24T14:14:51.956 INFO:teuthology.orchestra.run.smithi123.stdout: udica noarch 0.2.6-20.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 49 k 2024-01-24T14:14:51.956 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-01-24T14:14:51.956 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2024-01-24T14:14:51.957 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2024-01-24T14:14:51.957 INFO:teuthology.orchestra.run.smithi123.stdout: yajl x86_64 2.1.0-12.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2024-01-24T14:14:51.957 INFO:teuthology.orchestra.run.smithi123.stdout:Installing weak dependencies: 2024-01-24T14:14:51.957 INFO:teuthology.orchestra.run.smithi123.stdout: criu-libs x86_64 3.18-4.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 39 k 2024-01-24T14:14:51.957 INFO:teuthology.orchestra.run.smithi123.stdout:Installing module profiles: 2024-01-24T14:14:51.957 INFO:teuthology.orchestra.run.smithi123.stdout: container-tools/common 2024-01-24T14:14:51.957 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-24T14:14:51.957 INFO:teuthology.orchestra.run.smithi123.stdout:Enabling module streams: 2024-01-24T14:14:51.957 INFO:teuthology.orchestra.run.smithi123.stdout: container-tools rhel8 2024-01-24T14:14:51.957 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-24T14:14:51.957 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-01-24T14:14:51.957 INFO:teuthology.orchestra.run.smithi123.stdout:========================================================================================================== 2024-01-24T14:14:51.957 INFO:teuthology.orchestra.run.smithi123.stdout:Install 13 Packages 2024-01-24T14:14:51.958 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-24T14:14:51.958 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 26 M 2024-01-24T14:14:51.958 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 87 M 2024-01-24T14:14:51.959 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-01-24T14:14:52.021 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 6/13 2024-01-24T14:14:52.181 INFO:teuthology.orchestra.run.smithi123.stdout:(1/13): python3-pytoml-0.1.14-5.git7dea353.el8. 111 kB/s | 25 kB 00:00 2024-01-24T14:14:52.214 INFO:teuthology.orchestra.run.smithi123.stdout:(2/13): yajl-2.1.0-12.el8.x86_64.rpm 160 kB/s | 41 kB 00:00 2024-01-24T14:14:52.239 INFO:teuthology.orchestra.run.smithi123.stdout:(3/13): python3-pyxdg-0.25-16.el8.noarch.rpm 334 kB/s | 94 kB 00:00 2024-01-24T14:14:52.250 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-podman-4.6.0-1.module+el8.9.0+20326+387084 7/13 2024-01-24T14:14:52.390 INFO:teuthology.orchestra.run.smithi123.stdout:(4/13): aardvark-dns-1.7.0-1.module+el8.9.0+203 4.8 MB/s | 1.0 MB 00:00 2024-01-24T14:14:52.451 INFO:teuthology.orchestra.run.smithi123.stdout:(5/13): cockpit-podman-75-1.module+el8.9.0+2032 3.4 MB/s | 739 kB 00:00 2024-01-24T14:14:52.540 INFO:teuthology.orchestra.run.smithi123.stdout:(6/13): criu-libs-3.18-4.module+el8.9.0+20326+3 262 kB/s | 39 kB 00:00 2024-01-24T14:14:52.674 INFO:teuthology.orchestra.run.smithi123.stdout:(7/13): buildah-1.31.3-1.module+el8.9.0+20326+3 19 MB/s | 8.8 MB 00:00 2024-01-24T14:14:52.721 INFO:teuthology.orchestra.run.smithi123.stdout:(8/13): crun-1.8.7-1.module+el8.9.0+20326+38708 886 kB/s | 239 kB 00:00 2024-01-24T14:14:52.816 INFO:teuthology.orchestra.run.smithi123.stdout:(9/13): netavark-1.7.0-1.module+el8.9.0+20326+3 13 MB/s | 3.7 MB 00:00 2024-01-24T14:14:52.841 INFO:teuthology.orchestra.run.smithi123.stdout:(10/13): python3-podman-4.6.0-1.module+el8.9.0+ 918 kB/s | 153 kB 00:00 2024-01-24T14:14:53.008 INFO:teuthology.orchestra.run.smithi123.stdout:(11/13): toolbox-0.0.99.4-5.module+el8.9.0+2032 13 MB/s | 2.5 MB 00:00 2024-01-24T14:14:53.025 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-01-24T14:14:53.041 INFO:teuthology.orchestra.run.smithi123.stdout:(12/13): udica-0.2.6-20.module+el8.9.0+20326+38 247 kB/s | 49 kB 00:00 2024-01-24T14:14:53.142 INFO:teuthology.orchestra.run.smithi123.stdout:(13/13): skopeo-1.13.3-1.module+el8.9.0+20326+3 19 MB/s | 8.1 MB 00:00 2024-01-24T14:14:53.143 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-01-24T14:14:53.143 INFO:teuthology.orchestra.run.smithi123.stdout:Total 22 MB/s | 26 MB 00:01 2024-01-24T14:14:53.170 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : crun-1.8.7-1.module+el8.9.0+20326+387084d0.x86_64 8/13 2024-01-24T14:14:53.195 INFO:teuthology.orchestra.run.smithi019.stdout:========================================================================================================== 2024-01-24T14:14:53.195 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-01-24T14:14:53.196 INFO:teuthology.orchestra.run.smithi019.stdout:========================================================================================================== 2024-01-24T14:14:53.196 INFO:teuthology.orchestra.run.smithi019.stdout:Installing group/module packages: 2024-01-24T14:14:53.196 INFO:teuthology.orchestra.run.smithi019.stdout: aardvark-dns x86_64 2:1.7.0-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 1.0 M 2024-01-24T14:14:53.196 INFO:teuthology.orchestra.run.smithi019.stdout: buildah x86_64 1:1.31.3-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 8.8 M 2024-01-24T14:14:53.196 INFO:teuthology.orchestra.run.smithi019.stdout: cockpit-podman noarch 75-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 739 k 2024-01-24T14:14:53.196 INFO:teuthology.orchestra.run.smithi019.stdout: crun x86_64 1.8.7-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 239 k 2024-01-24T14:14:53.196 INFO:teuthology.orchestra.run.smithi019.stdout: netavark x86_64 2:1.7.0-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-01-24T14:14:53.196 INFO:teuthology.orchestra.run.smithi019.stdout: python3-podman noarch 4.6.0-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 153 k 2024-01-24T14:14:53.196 INFO:teuthology.orchestra.run.smithi019.stdout: skopeo x86_64 2:1.13.3-1.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 8.1 M 2024-01-24T14:14:53.196 INFO:teuthology.orchestra.run.smithi019.stdout: toolbox x86_64 0.0.99.4-5.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 2.5 M 2024-01-24T14:14:53.196 INFO:teuthology.orchestra.run.smithi019.stdout: udica noarch 0.2.6-20.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 49 k 2024-01-24T14:14:53.196 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-01-24T14:14:53.197 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2024-01-24T14:14:53.197 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2024-01-24T14:14:53.197 INFO:teuthology.orchestra.run.smithi019.stdout: yajl x86_64 2.1.0-12.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2024-01-24T14:14:53.197 INFO:teuthology.orchestra.run.smithi019.stdout:Installing weak dependencies: 2024-01-24T14:14:53.197 INFO:teuthology.orchestra.run.smithi019.stdout: criu-libs x86_64 3.18-4.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 39 k 2024-01-24T14:14:53.197 INFO:teuthology.orchestra.run.smithi019.stdout:Installing module profiles: 2024-01-24T14:14:53.197 INFO:teuthology.orchestra.run.smithi019.stdout: container-tools/common 2024-01-24T14:14:53.197 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:14:53.197 INFO:teuthology.orchestra.run.smithi019.stdout:Enabling module streams: 2024-01-24T14:14:53.197 INFO:teuthology.orchestra.run.smithi019.stdout: container-tools rhel8 2024-01-24T14:14:53.197 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:14:53.197 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-01-24T14:14:53.197 INFO:teuthology.orchestra.run.smithi019.stdout:========================================================================================================== 2024-01-24T14:14:53.197 INFO:teuthology.orchestra.run.smithi019.stdout:Install 13 Packages 2024-01-24T14:14:53.198 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:14:53.198 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 26 M 2024-01-24T14:14:53.198 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 87 M 2024-01-24T14:14:53.198 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-01-24T14:14:53.382 INFO:teuthology.orchestra.run.smithi019.stdout:(1/13): python3-pyxdg-0.25-16.el8.noarch.rpm 514 kB/s | 94 kB 00:00 2024-01-24T14:14:53.407 INFO:teuthology.orchestra.run.smithi019.stdout:(2/13): python3-pytoml-0.1.14-5.git7dea353.el8. 119 kB/s | 25 kB 00:00 2024-01-24T14:14:53.414 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : buildah-1:1.31.3-1.module+el8.9.0+20326+387084d0.x 9/13 2024-01-24T14:14:53.425 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-01-24T14:14:53.446 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-01-24T14:14:53.447 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-01-24T14:14:53.449 INFO:teuthology.orchestra.run.smithi019.stdout:(3/13): yajl-2.1.0-12.el8.x86_64.rpm 164 kB/s | 41 kB 00:00 2024-01-24T14:14:53.508 INFO:teuthology.orchestra.run.smithi019.stdout:(4/13): cockpit-podman-75-1.module+el8.9.0+2032 12 MB/s | 739 kB 00:00 2024-01-24T14:14:53.549 INFO:teuthology.orchestra.run.smithi019.stdout:(5/13): aardvark-dns-1.7.0-1.module+el8.9.0+203 6.0 MB/s | 1.0 MB 00:00 2024-01-24T14:14:53.591 INFO:teuthology.orchestra.run.smithi019.stdout:(6/13): criu-libs-3.18-4.module+el8.9.0+20326+3 472 kB/s | 39 kB 00:00 2024-01-24T14:14:53.675 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-01-24T14:14:53.684 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-01-24T14:14:53.775 INFO:teuthology.orchestra.run.smithi019.stdout:(7/13): buildah-1.31.3-1.module+el8.9.0+20326+3 24 MB/s | 8.8 MB 00:00 2024-01-24T14:14:53.808 INFO:teuthology.orchestra.run.smithi019.stdout:(8/13): crun-1.8.7-1.module+el8.9.0+20326+38708 924 kB/s | 239 kB 00:00 2024-01-24T14:14:53.817 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : udica-0.2.6-20.module+el8.9.0+20326+387084d0.noarc 10/13 2024-01-24T14:14:53.909 INFO:teuthology.orchestra.run.smithi019.stdout:(9/13): netavark-1.7.0-1.module+el8.9.0+20326+3 12 MB/s | 3.7 MB 00:00 2024-01-24T14:14:53.934 INFO:teuthology.orchestra.run.smithi019.stdout:(10/13): python3-podman-4.6.0-1.module+el8.9.0+ 969 kB/s | 153 kB 00:00 2024-01-24T14:14:54.051 INFO:teuthology.orchestra.run.smithi019.stdout:(11/13): udica-0.2.6-20.module+el8.9.0+20326+38 423 kB/s | 49 kB 00:00 2024-01-24T14:14:54.150 INFO:teuthology.orchestra.run.smithi019.stdout:(12/13): toolbox-0.0.99.4-5.module+el8.9.0+2032 11 MB/s | 2.5 MB 00:00 2024-01-24T14:14:54.251 INFO:teuthology.orchestra.run.smithi019.stdout:(13/13): skopeo-1.13.3-1.module+el8.9.0+20326+3 18 MB/s | 8.1 MB 00:00 2024-01-24T14:14:54.252 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-01-24T14:14:54.252 INFO:teuthology.orchestra.run.smithi019.stdout:Total 24 MB/s | 26 MB 00:01 2024-01-24T14:14:54.295 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-01-24T14:14:54.562 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-01-24T14:14:54.564 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 1/13 2024-01-24T14:14:54.586 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-01-24T14:14:54.586 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-01-24T14:14:54.699 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : toolbox-0.0.99.4-5.module+el8.9.0+20326+387084d0.x 11/13 2024-01-24T14:14:54.841 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-01-24T14:14:54.851 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-01-24T14:14:54.929 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : skopeo-2:1.13.3-1.module+el8.9.0+20326+387084d0.x8 12/13 2024-01-24T14:14:55.084 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : aardvark-dns-2:1.7.0-1.module+el8.9.0+20326+387084 2/13 2024-01-24T14:14:55.128 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : cockpit-podman-75-1.module+el8.9.0+20326+387084d0. 13/13 2024-01-24T14:14:55.293 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : netavark-2:1.7.0-1.module+el8.9.0+20326+387084d0.x 3/13 2024-01-24T14:14:55.499 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-01-24T14:14:55.500 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : yajl-2.1.0-12.el8.x86_64 4/13 2024-01-24T14:14:55.543 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: cockpit-podman-75-1.module+el8.9.0+20326+387084d0. 13/13 2024-01-24T14:14:55.544 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/13 2024-01-24T14:14:55.544 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-01-24T14:14:55.544 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 3/13 2024-01-24T14:14:55.544 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : aardvark-dns-2:1.7.0-1.module+el8.9.0+20326+387084 4/13 2024-01-24T14:14:55.545 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : buildah-1:1.31.3-1.module+el8.9.0+20326+387084d0.x 5/13 2024-01-24T14:14:55.545 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : cockpit-podman-75-1.module+el8.9.0+20326+387084d0. 6/13 2024-01-24T14:14:55.545 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 7/13 2024-01-24T14:14:55.545 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : crun-1.8.7-1.module+el8.9.0+20326+387084d0.x86_64 8/13 2024-01-24T14:14:55.545 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : netavark-2:1.7.0-1.module+el8.9.0+20326+387084d0.x 9/13 2024-01-24T14:14:55.545 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-podman-4.6.0-1.module+el8.9.0+20326+387084 10/13 2024-01-24T14:14:55.546 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : skopeo-2:1.13.3-1.module+el8.9.0+20326+387084d0.x8 11/13 2024-01-24T14:14:55.546 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : toolbox-0.0.99.4-5.module+el8.9.0+20326+387084d0.x 12/13 2024-01-24T14:14:55.661 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 5/13 2024-01-24T14:14:55.730 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 1/13 2024-01-24T14:14:55.929 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 6/13 2024-01-24T14:14:56.215 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-podman-4.6.0-1.module+el8.9.0+20326+387084 7/13 2024-01-24T14:14:56.269 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : aardvark-dns-2:1.7.0-1.module+el8.9.0+20326+387084 2/13 2024-01-24T14:14:56.444 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : netavark-2:1.7.0-1.module+el8.9.0+20326+387084d0.x 3/13 2024-01-24T14:14:56.643 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : yajl-2.1.0-12.el8.x86_64 4/13 2024-01-24T14:14:56.802 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 5/13 2024-01-24T14:14:57.062 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 6/13 2024-01-24T14:14:57.111 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : crun-1.8.7-1.module+el8.9.0+20326+387084d0.x86_64 8/13 2024-01-24T14:14:57.306 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : buildah-1:1.31.3-1.module+el8.9.0+20326+387084d0.x 9/13 2024-01-24T14:14:57.306 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-podman-4.6.0-1.module+el8.9.0+20326+387084 7/13 2024-01-24T14:14:57.697 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : udica-0.2.6-20.module+el8.9.0+20326+387084d0.noarc 10/13 2024-01-24T14:14:58.275 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : crun-1.8.7-1.module+el8.9.0+20326+387084d0.x86_64 8/13 2024-01-24T14:14:58.509 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : buildah-1:1.31.3-1.module+el8.9.0+20326+387084d0.x 9/13 2024-01-24T14:14:58.579 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : toolbox-0.0.99.4-5.module+el8.9.0+20326+387084d0.x 11/13 2024-01-24T14:14:58.851 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : skopeo-2:1.13.3-1.module+el8.9.0+20326+387084d0.x8 12/13 2024-01-24T14:14:58.881 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : udica-0.2.6-20.module+el8.9.0+20326+387084d0.noarc 10/13 2024-01-24T14:14:59.060 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : cockpit-podman-75-1.module+el8.9.0+20326+387084d0. 13/13 2024-01-24T14:14:59.468 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: cockpit-podman-75-1.module+el8.9.0+20326+387084d0. 13/13 2024-01-24T14:14:59.468 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/13 2024-01-24T14:14:59.468 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-01-24T14:14:59.468 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 3/13 2024-01-24T14:14:59.468 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : aardvark-dns-2:1.7.0-1.module+el8.9.0+20326+387084 4/13 2024-01-24T14:14:59.468 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : buildah-1:1.31.3-1.module+el8.9.0+20326+387084d0.x 5/13 2024-01-24T14:14:59.468 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : cockpit-podman-75-1.module+el8.9.0+20326+387084d0. 6/13 2024-01-24T14:14:59.468 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 7/13 2024-01-24T14:14:59.468 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : crun-1.8.7-1.module+el8.9.0+20326+387084d0.x86_64 8/13 2024-01-24T14:14:59.468 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : netavark-2:1.7.0-1.module+el8.9.0+20326+387084d0.x 9/13 2024-01-24T14:14:59.469 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-podman-4.6.0-1.module+el8.9.0+20326+387084 10/13 2024-01-24T14:14:59.469 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : skopeo-2:1.13.3-1.module+el8.9.0+20326+387084d0.x8 11/13 2024-01-24T14:14:59.469 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : toolbox-0.0.99.4-5.module+el8.9.0+20326+387084d0.x 12/13 2024-01-24T14:14:59.828 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : toolbox-0.0.99.4-5.module+el8.9.0+20326+387084d0.x 11/13 2024-01-24T14:15:00.180 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : skopeo-2:1.13.3-1.module+el8.9.0+20326+387084d0.x8 12/13 2024-01-24T14:15:00.333 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : cockpit-podman-75-1.module+el8.9.0+20326+387084d0. 13/13 2024-01-24T14:15:00.813 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: cockpit-podman-75-1.module+el8.9.0+20326+387084d0. 13/13 2024-01-24T14:15:00.814 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/13 2024-01-24T14:15:00.814 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-01-24T14:15:00.814 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 3/13 2024-01-24T14:15:00.814 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : aardvark-dns-2:1.7.0-1.module+el8.9.0+20326+387084 4/13 2024-01-24T14:15:00.814 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : buildah-1:1.31.3-1.module+el8.9.0+20326+387084d0.x 5/13 2024-01-24T14:15:00.814 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : cockpit-podman-75-1.module+el8.9.0+20326+387084d0. 6/13 2024-01-24T14:15:00.814 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 7/13 2024-01-24T14:15:00.814 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : crun-1.8.7-1.module+el8.9.0+20326+387084d0.x86_64 8/13 2024-01-24T14:15:00.814 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : netavark-2:1.7.0-1.module+el8.9.0+20326+387084d0.x 9/13 2024-01-24T14:15:00.814 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-podman-4.6.0-1.module+el8.9.0+20326+387084 10/13 2024-01-24T14:15:00.814 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : skopeo-2:1.13.3-1.module+el8.9.0+20326+387084d0.x8 11/13 2024-01-24T14:15:00.814 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : toolbox-0.0.99.4-5.module+el8.9.0+20326+387084d0.x 12/13 2024-01-24T14:15:07.210 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : udica-0.2.6-20.module+el8.9.0+20326+387084d0.noarc 13/13 2024-01-24T14:15:07.210 INFO:teuthology.orchestra.run.smithi194.stdout:Installed products updated. 2024-01-24T14:15:07.764 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-24T14:15:07.764 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-01-24T14:15:07.764 INFO:teuthology.orchestra.run.smithi194.stdout: aardvark-dns-2:1.7.0-1.module+el8.9.0+20326+387084d0.x86_64 2024-01-24T14:15:07.764 INFO:teuthology.orchestra.run.smithi194.stdout: buildah-1:1.31.3-1.module+el8.9.0+20326+387084d0.x86_64 2024-01-24T14:15:07.764 INFO:teuthology.orchestra.run.smithi194.stdout: cockpit-podman-75-1.module+el8.9.0+20326+387084d0.noarch 2024-01-24T14:15:07.765 INFO:teuthology.orchestra.run.smithi194.stdout: criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86_64 2024-01-24T14:15:07.765 INFO:teuthology.orchestra.run.smithi194.stdout: crun-1.8.7-1.module+el8.9.0+20326+387084d0.x86_64 2024-01-24T14:15:07.765 INFO:teuthology.orchestra.run.smithi194.stdout: netavark-2:1.7.0-1.module+el8.9.0+20326+387084d0.x86_64 2024-01-24T14:15:07.765 INFO:teuthology.orchestra.run.smithi194.stdout: python3-podman-4.6.0-1.module+el8.9.0+20326+387084d0.noarch 2024-01-24T14:15:07.765 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-01-24T14:15:07.765 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-01-24T14:15:07.765 INFO:teuthology.orchestra.run.smithi194.stdout: skopeo-2:1.13.3-1.module+el8.9.0+20326+387084d0.x86_64 2024-01-24T14:15:07.765 INFO:teuthology.orchestra.run.smithi194.stdout: toolbox-0.0.99.4-5.module+el8.9.0+20326+387084d0.x86_64 2024-01-24T14:15:07.765 INFO:teuthology.orchestra.run.smithi194.stdout: udica-0.2.6-20.module+el8.9.0+20326+387084d0.noarch 2024-01-24T14:15:07.765 INFO:teuthology.orchestra.run.smithi194.stdout: yajl-2.1.0-12.el8.x86_64 2024-01-24T14:15:07.765 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-24T14:15:07.765 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-24T14:15:07.930 DEBUG:teuthology.parallel:result is None 2024-01-24T14:15:09.182 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : udica-0.2.6-20.module+el8.9.0+20326+387084d0.noarc 13/13 2024-01-24T14:15:09.182 INFO:teuthology.orchestra.run.smithi123.stdout:Installed products updated. 2024-01-24T14:15:09.759 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-24T14:15:09.759 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-01-24T14:15:09.759 INFO:teuthology.orchestra.run.smithi123.stdout: aardvark-dns-2:1.7.0-1.module+el8.9.0+20326+387084d0.x86_64 2024-01-24T14:15:09.759 INFO:teuthology.orchestra.run.smithi123.stdout: buildah-1:1.31.3-1.module+el8.9.0+20326+387084d0.x86_64 2024-01-24T14:15:09.759 INFO:teuthology.orchestra.run.smithi123.stdout: cockpit-podman-75-1.module+el8.9.0+20326+387084d0.noarch 2024-01-24T14:15:09.759 INFO:teuthology.orchestra.run.smithi123.stdout: criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86_64 2024-01-24T14:15:09.760 INFO:teuthology.orchestra.run.smithi123.stdout: crun-1.8.7-1.module+el8.9.0+20326+387084d0.x86_64 2024-01-24T14:15:09.760 INFO:teuthology.orchestra.run.smithi123.stdout: netavark-2:1.7.0-1.module+el8.9.0+20326+387084d0.x86_64 2024-01-24T14:15:09.760 INFO:teuthology.orchestra.run.smithi123.stdout: python3-podman-4.6.0-1.module+el8.9.0+20326+387084d0.noarch 2024-01-24T14:15:09.760 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-01-24T14:15:09.760 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-01-24T14:15:09.760 INFO:teuthology.orchestra.run.smithi123.stdout: skopeo-2:1.13.3-1.module+el8.9.0+20326+387084d0.x86_64 2024-01-24T14:15:09.760 INFO:teuthology.orchestra.run.smithi123.stdout: toolbox-0.0.99.4-5.module+el8.9.0+20326+387084d0.x86_64 2024-01-24T14:15:09.760 INFO:teuthology.orchestra.run.smithi123.stdout: udica-0.2.6-20.module+el8.9.0+20326+387084d0.noarch 2024-01-24T14:15:09.760 INFO:teuthology.orchestra.run.smithi123.stdout: yajl-2.1.0-12.el8.x86_64 2024-01-24T14:15:09.760 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-24T14:15:09.760 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-01-24T14:15:09.950 DEBUG:teuthology.parallel:result is None 2024-01-24T14:15:13.551 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : udica-0.2.6-20.module+el8.9.0+20326+387084d0.noarc 13/13 2024-01-24T14:15:13.551 INFO:teuthology.orchestra.run.smithi019.stdout:Installed products updated. 2024-01-24T14:15:14.161 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:15:14.161 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-01-24T14:15:14.161 INFO:teuthology.orchestra.run.smithi019.stdout: aardvark-dns-2:1.7.0-1.module+el8.9.0+20326+387084d0.x86_64 2024-01-24T14:15:14.161 INFO:teuthology.orchestra.run.smithi019.stdout: buildah-1:1.31.3-1.module+el8.9.0+20326+387084d0.x86_64 2024-01-24T14:15:14.161 INFO:teuthology.orchestra.run.smithi019.stdout: cockpit-podman-75-1.module+el8.9.0+20326+387084d0.noarch 2024-01-24T14:15:14.161 INFO:teuthology.orchestra.run.smithi019.stdout: criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86_64 2024-01-24T14:15:14.161 INFO:teuthology.orchestra.run.smithi019.stdout: crun-1.8.7-1.module+el8.9.0+20326+387084d0.x86_64 2024-01-24T14:15:14.161 INFO:teuthology.orchestra.run.smithi019.stdout: netavark-2:1.7.0-1.module+el8.9.0+20326+387084d0.x86_64 2024-01-24T14:15:14.162 INFO:teuthology.orchestra.run.smithi019.stdout: python3-podman-4.6.0-1.module+el8.9.0+20326+387084d0.noarch 2024-01-24T14:15:14.162 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-01-24T14:15:14.162 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-01-24T14:15:14.162 INFO:teuthology.orchestra.run.smithi019.stdout: skopeo-2:1.13.3-1.module+el8.9.0+20326+387084d0.x86_64 2024-01-24T14:15:14.162 INFO:teuthology.orchestra.run.smithi019.stdout: toolbox-0.0.99.4-5.module+el8.9.0+20326+387084d0.x86_64 2024-01-24T14:15:14.162 INFO:teuthology.orchestra.run.smithi019.stdout: udica-0.2.6-20.module+el8.9.0+20326+387084d0.noarch 2024-01-24T14:15:14.162 INFO:teuthology.orchestra.run.smithi019.stdout: yajl-2.1.0-12.el8.x86_64 2024-01-24T14:15:14.162 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:15:14.162 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-01-24T14:15:14.359 DEBUG:teuthology.parallel:result is None 2024-01-24T14:15:14.359 INFO:teuthology.run_tasks:Running task install... 2024-01-24T14:15:14.370 DEBUG:teuthology.task.install:project ceph 2024-01-24T14:15:14.370 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '4e3182a65800aed6d4aa2f6b86da3cfd72188134'}} 2024-01-24T14:15:14.370 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '4e3182a65800aed6d4aa2f6b86da3cfd72188134'} 2024-01-24T14:15:14.370 INFO:teuthology.task.install:Using flavor: default 2024-01-24T14:15:14.378 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-01-24T14:15:14.379 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2024-01-24T14:15:14.379 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '4e3182a65800aed6d4aa2f6b86da3cfd72188134', 'tag': None, 'wait_for_package': False} 2024-01-24T14:15:14.379 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4e3182a65800aed6d4aa2f6b86da3cfd72188134 2024-01-24T14:15:14.382 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '4e3182a65800aed6d4aa2f6b86da3cfd72188134', 'tag': None, 'wait_for_package': False} 2024-01-24T14:15:14.382 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4e3182a65800aed6d4aa2f6b86da3cfd72188134 2024-01-24T14:15:14.383 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '4e3182a65800aed6d4aa2f6b86da3cfd72188134', 'tag': None, 'wait_for_package': False} 2024-01-24T14:15:14.383 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4e3182a65800aed6d4aa2f6b86da3cfd72188134 2024-01-24T14:15:14.534 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing1-reef-2024-01-24-1150/4e3182a65800aed6d4aa2f6b86da3cfd72188134/centos/8/flavors/default/ 2024-01-24T14:15:14.534 INFO:teuthology.task.install.rpm:Package version is 18.2.1-341.g4e3182a6 2024-01-24T14:15:14.601 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing1-reef-2024-01-24-1150/4e3182a65800aed6d4aa2f6b86da3cfd72188134/centos/8/flavors/default/ 2024-01-24T14:15:14.601 INFO:teuthology.task.install.rpm:Package version is 18.2.1-341.g4e3182a6 2024-01-24T14:15:14.604 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing1-reef-2024-01-24-1150/4e3182a65800aed6d4aa2f6b86da3cfd72188134/centos/8/flavors/default/ 2024-01-24T14:15:14.604 INFO:teuthology.task.install.rpm:Package version is 18.2.1-341.g4e3182a6 2024-01-24T14:15:14.652 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing1-reef-2024-01-24-1150/4e3182a65800aed6d4aa2f6b86da3cfd72188134/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing1-reef-2024-01-24-1150/4e3182a65800aed6d4aa2f6b86da3cfd72188134/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing1-reef-2024-01-24-1150/4e3182a65800aed6d4aa2f6b86da3cfd72188134/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-24T14:15:14.652 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-01-24T14:15:14.652 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-24T14:15:14.688 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-01-24T14:15:14.688 DEBUG:teuthology.orchestra.run.smithi019:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4e3182a65800aed6d4aa2f6b86da3cfd72188134/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-24T14:15:14.710 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing1-reef-2024-01-24-1150/4e3182a65800aed6d4aa2f6b86da3cfd72188134/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing1-reef-2024-01-24-1150/4e3182a65800aed6d4aa2f6b86da3cfd72188134/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing1-reef-2024-01-24-1150/4e3182a65800aed6d4aa2f6b86da3cfd72188134/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-24T14:15:14.710 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-01-24T14:15:14.710 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-24T14:15:14.716 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing1-reef-2024-01-24-1150/4e3182a65800aed6d4aa2f6b86da3cfd72188134/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing1-reef-2024-01-24-1150/4e3182a65800aed6d4aa2f6b86da3cfd72188134/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing1-reef-2024-01-24-1150/4e3182a65800aed6d4aa2f6b86da3cfd72188134/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-24T14:15:14.716 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-01-24T14:15:14.716 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-24T14:15:14.745 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-01-24T14:15:14.745 DEBUG:teuthology.orchestra.run.smithi123:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4e3182a65800aed6d4aa2f6b86da3cfd72188134/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-24T14:15:14.748 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-01-24T14:15:14.749 DEBUG:teuthology.orchestra.run.smithi194:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4e3182a65800aed6d4aa2f6b86da3cfd72188134/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-24T14:15:14.764 DEBUG:teuthology.orchestra.run.smithi019:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-24T14:15:14.819 DEBUG:teuthology.orchestra.run.smithi123:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-24T14:15:14.820 DEBUG:teuthology.orchestra.run.smithi194:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-24T14:15:14.851 DEBUG:teuthology.orchestra.run.smithi019:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-24T14:15:14.883 INFO:teuthology.orchestra.run.smithi019.stdout:check_obsoletes = 1 2024-01-24T14:15:14.885 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum clean all 2024-01-24T14:15:14.902 DEBUG:teuthology.orchestra.run.smithi194:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-24T14:15:14.905 DEBUG:teuthology.orchestra.run.smithi123:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-24T14:15:14.937 INFO:teuthology.orchestra.run.smithi123.stdout:check_obsoletes = 1 2024-01-24T14:15:14.939 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum clean all 2024-01-24T14:15:14.972 INFO:teuthology.orchestra.run.smithi194.stdout:check_obsoletes = 1 2024-01-24T14:15:14.973 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum clean all 2024-01-24T14:15:15.320 INFO:teuthology.orchestra.run.smithi019.stdout:Updating Subscription Management repositories. 2024-01-24T14:15:15.332 INFO:teuthology.orchestra.run.smithi123.stdout:Updating Subscription Management repositories. 2024-01-24T14:15:15.350 INFO:teuthology.orchestra.run.smithi194.stdout:Updating Subscription Management repositories. 2024-01-24T14:15:16.652 INFO:teuthology.orchestra.run.smithi194.stdout:57 files removed 2024-01-24T14:15:16.702 INFO:teuthology.orchestra.run.smithi019.stdout:57 files removed 2024-01-24T14:15:16.703 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-01-24T14:15:16.714 INFO:teuthology.orchestra.run.smithi123.stdout:57 files removed 2024-01-24T14:15:16.758 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-01-24T14:15:16.765 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-01-24T14:15:17.040 INFO:teuthology.orchestra.run.smithi194.stdout:Updating Subscription Management repositories. 2024-01-24T14:15:17.102 INFO:teuthology.orchestra.run.smithi123.stdout:Updating Subscription Management repositories. 2024-01-24T14:15:17.144 INFO:teuthology.orchestra.run.smithi019.stdout:Updating Subscription Management repositories. 2024-01-24T14:15:18.871 INFO:teuthology.orchestra.run.smithi123.stdout:ceph packages for x86_64 194 kB/s | 80 kB 00:00 2024-01-24T14:15:18.877 INFO:teuthology.orchestra.run.smithi194.stdout:ceph packages for x86_64 235 kB/s | 80 kB 00:00 2024-01-24T14:15:19.141 INFO:teuthology.orchestra.run.smithi019.stdout:ceph packages for x86_64 223 kB/s | 80 kB 00:00 2024-01-24T14:15:19.144 INFO:teuthology.orchestra.run.smithi194.stdout:ceph noarch packages 41 kB/s | 9.5 kB 00:00 2024-01-24T14:15:19.180 INFO:teuthology.orchestra.run.smithi123.stdout:ceph noarch packages 35 kB/s | 9.5 kB 00:00 2024-01-24T14:15:19.467 INFO:teuthology.orchestra.run.smithi019.stdout:ceph noarch packages 33 kB/s | 9.5 kB 00:00 2024-01-24T14:15:19.470 INFO:teuthology.orchestra.run.smithi194.stdout:ceph source packages 5.0 kB/s | 1.5 kB 00:00 2024-01-24T14:15:19.481 INFO:teuthology.orchestra.run.smithi123.stdout:ceph source packages 5.4 kB/s | 1.5 kB 00:00 2024-01-24T14:15:19.657 INFO:teuthology.orchestra.run.smithi123.stdout:Copr repo for python3-asyncssh owned by ceph 23 kB/s | 3.5 kB 00:00 2024-01-24T14:15:19.681 INFO:teuthology.orchestra.run.smithi194.stdout:Copr repo for python3-asyncssh owned by ceph 18 kB/s | 3.5 kB 00:00 2024-01-24T14:15:19.818 INFO:teuthology.orchestra.run.smithi019.stdout:ceph source packages 4.6 kB/s | 1.5 kB 00:00 2024-01-24T14:15:19.985 INFO:teuthology.orchestra.run.smithi019.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2024-01-24T14:15:20.507 INFO:teuthology.orchestra.run.smithi123.stdout:Extra Packages for Enterprise Linux 20 MB/s | 16 MB 00:00 2024-01-24T14:15:20.546 INFO:teuthology.orchestra.run.smithi194.stdout:Extra Packages for Enterprise Linux 19 MB/s | 16 MB 00:00 2024-01-24T14:15:23.739 INFO:teuthology.orchestra.run.smithi019.stdout:Extra Packages for Enterprise Linux 4.3 MB/s | 16 MB 00:03 2024-01-24T14:15:23.999 INFO:teuthology.orchestra.run.smithi123.stdout:lab-extras 550 kB/s | 24 kB 00:00 2024-01-24T14:15:24.189 INFO:teuthology.orchestra.run.smithi194.stdout:lab-extras 404 kB/s | 24 kB 00:00 2024-01-24T14:15:25.476 INFO:teuthology.orchestra.run.smithi123.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 46 MB/s | 65 MB 00:01 2024-01-24T14:15:25.623 INFO:teuthology.orchestra.run.smithi194.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 43 MB/s | 59 MB 00:01 2024-01-24T14:15:27.761 INFO:teuthology.orchestra.run.smithi019.stdout:lab-extras 436 kB/s | 24 kB 00:00 2024-01-24T14:15:28.455 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 14 MB/s | 8.8 MB 00:00 2024-01-24T14:15:31.490 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 44 MB/s | 59 MB 00:01 2024-01-24T14:15:36.010 INFO:teuthology.orchestra.run.smithi123.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 47 MB/s | 59 MB 00:01 2024-01-24T14:15:36.182 INFO:teuthology.orchestra.run.smithi194.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 16 MB/s | 8.8 MB 00:00 2024-01-24T14:15:39.050 INFO:teuthology.orchestra.run.smithi194.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 52 MB/s | 65 MB 00:01 2024-01-24T14:15:43.866 INFO:teuthology.orchestra.run.smithi019.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 48 MB/s | 65 MB 00:01 2024-01-24T14:15:46.468 INFO:teuthology.orchestra.run.smithi123.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 14 MB/s | 8.8 MB 00:00 2024-01-24T14:15:53.567 INFO:teuthology.orchestra.run.smithi123.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-01-24T14:15:53.568 INFO:teuthology.orchestra.run.smithi123.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-01-24T14:15:53.588 INFO:teuthology.orchestra.run.smithi123.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2024-01-24T14:15:53.589 INFO:teuthology.orchestra.run.smithi123.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2024-01-24T14:15:53.591 INFO:teuthology.orchestra.run.smithi123.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2024-01-24T14:15:53.592 INFO:teuthology.orchestra.run.smithi123.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2024-01-24T14:15:53.595 INFO:teuthology.orchestra.run.smithi123.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2024-01-24T14:15:53.599 INFO:teuthology.orchestra.run.smithi123.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2024-01-24T14:15:53.601 INFO:teuthology.orchestra.run.smithi123.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2024-01-24T14:15:53.811 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-01-24T14:15:53.825 INFO:teuthology.orchestra.run.smithi123.stdout:=========================================================================================================================== 2024-01-24T14:15:53.825 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2024-01-24T14:15:53.825 INFO:teuthology.orchestra.run.smithi123.stdout:=========================================================================================================================== 2024-01-24T14:15:53.825 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-01-24T14:15:53.825 INFO:teuthology.orchestra.run.smithi123.stdout: NetworkManager-initscripts-updown noarch 1:1.40.16-13.el8_9 rhel-8-for-x86_64-baseos-rpms 144 k 2024-01-24T14:15:53.825 INFO:teuthology.orchestra.run.smithi123.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2024-01-24T14:15:53.825 INFO:teuthology.orchestra.run.smithi123.stdout: ceph x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 6.4 k 2024-01-24T14:15:53.826 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-base x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 5.4 M 2024-01-24T14:15:53.826 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-fuse x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 852 k 2024-01-24T14:15:53.826 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-immutable-object-cache x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 165 k 2024-01-24T14:15:53.826 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 1.6 M 2024-01-24T14:15:53.826 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-cephadm noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 132 k 2024-01-24T14:15:53.826 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-dashboard noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 1.8 M 2024-01-24T14:15:53.826 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 7.4 M 2024-01-24T14:15:53.826 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-rook noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 50 k 2024-01-24T14:15:53.826 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-radosgw x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 13 M 2024-01-24T14:15:53.826 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-test x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 47 M 2024-01-24T14:15:53.826 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-volume noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 262 k 2024-01-24T14:15:53.827 INFO:teuthology.orchestra.run.smithi123.stdout: cephadm noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 171 k 2024-01-24T14:15:53.827 INFO:teuthology.orchestra.run.smithi123.stdout: cephfs-mirror x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 259 k 2024-01-24T14:15:53.827 INFO:teuthology.orchestra.run.smithi123.stdout: cephfs-top noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 18 k 2024-01-24T14:15:53.827 INFO:teuthology.orchestra.run.smithi123.stdout: elfutils-libelf-devel x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 62 k 2024-01-24T14:15:53.827 INFO:teuthology.orchestra.run.smithi123.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2024-01-24T14:15:53.827 INFO:teuthology.orchestra.run.smithi123.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2024-01-24T14:15:53.827 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs-devel x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 31 k 2024-01-24T14:15:53.827 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs2 x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 776 k 2024-01-24T14:15:53.827 INFO:teuthology.orchestra.run.smithi123.stdout: librados-devel x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 139 k 2024-01-24T14:15:53.827 INFO:teuthology.orchestra.run.smithi123.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2024-01-24T14:15:53.827 INFO:teuthology.orchestra.run.smithi123.stdout: libuuid-devel x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 100 k 2024-01-24T14:15:53.827 INFO:teuthology.orchestra.run.smithi123.stdout: openssl-devel x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 2.3 M 2024-01-24T14:15:53.827 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql x86_64 10.23-3.module+el8.9.0+20831+78489779 rhel-8-for-x86_64-appstream-rpms 1.5 M 2024-01-24T14:15:53.828 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-contrib x86_64 10.23-3.module+el8.9.0+20831+78489779 rhel-8-for-x86_64-appstream-rpms 811 k 2024-01-24T14:15:53.828 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-server x86_64 10.23-3.module+el8.9.0+20831+78489779 rhel-8-for-x86_64-appstream-rpms 5.1 M 2024-01-24T14:15:53.828 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cephfs x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 197 k 2024-01-24T14:15:53.828 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rados x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 371 k 2024-01-24T14:15:53.828 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rbd x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 365 k 2024-01-24T14:15:53.828 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rgw x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 107 k 2024-01-24T14:15:53.828 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-fuse x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 94 k 2024-01-24T14:15:53.828 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-mirror x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 3.4 M 2024-01-24T14:15:53.828 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-nbd x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 181 k 2024-01-24T14:15:53.828 INFO:teuthology.orchestra.run.smithi123.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 73 k 2024-01-24T14:15:53.828 INFO:teuthology.orchestra.run.smithi123.stdout:Upgrading: 2024-01-24T14:15:53.828 INFO:teuthology.orchestra.run.smithi123.stdout: elfutils-libelf x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 232 k 2024-01-24T14:15:53.829 INFO:teuthology.orchestra.run.smithi123.stdout: elfutils-libs x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 303 k 2024-01-24T14:15:53.829 INFO:teuthology.orchestra.run.smithi123.stdout: iproute x86_64 6.2.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 854 k 2024-01-24T14:15:53.829 INFO:teuthology.orchestra.run.smithi123.stdout: libblkid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-01-24T14:15:53.829 INFO:teuthology.orchestra.run.smithi123.stdout: libfdisk x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 254 k 2024-01-24T14:15:53.829 INFO:teuthology.orchestra.run.smithi123.stdout: libmount x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 237 k 2024-01-24T14:15:53.829 INFO:teuthology.orchestra.run.smithi123.stdout: librados2 x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 3.7 M 2024-01-24T14:15:53.829 INFO:teuthology.orchestra.run.smithi123.stdout: librbd1 x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 3.6 M 2024-01-24T14:15:53.829 INFO:teuthology.orchestra.run.smithi123.stdout: libsmartcols x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 179 k 2024-01-24T14:15:53.829 INFO:teuthology.orchestra.run.smithi123.stdout: libuuid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 99 k 2024-01-24T14:15:53.829 INFO:teuthology.orchestra.run.smithi123.stdout: openssl x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 711 k 2024-01-24T14:15:53.829 INFO:teuthology.orchestra.run.smithi123.stdout: openssl-libs x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 1.5 M 2024-01-24T14:15:53.829 INFO:teuthology.orchestra.run.smithi123.stdout: util-linux x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 2.5 M 2024-01-24T14:15:53.829 INFO:teuthology.orchestra.run.smithi123.stdout: util-linux-user x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 103 k 2024-01-24T14:15:53.830 INFO:teuthology.orchestra.run.smithi123.stdout: xfsdump x86_64 3.1.8-7.el8_9 rhel-8-for-x86_64-baseos-rpms 333 k 2024-01-24T14:15:53.830 INFO:teuthology.orchestra.run.smithi123.stdout: xfsprogs x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M 2024-01-24T14:15:53.830 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-01-24T14:15:53.830 INFO:teuthology.orchestra.run.smithi123.stdout: automake noarch 1.16.1-8.el8 rhel-8-for-x86_64-appstream-rpms 714 k 2024-01-24T14:15:53.830 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-common x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 22 M 2024-01-24T14:15:53.830 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-grafana-dashboards noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 23 k 2024-01-24T14:15:53.830 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mds x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 2.3 M 2024-01-24T14:15:53.830 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-modules-core noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 242 k 2024-01-24T14:15:53.830 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mon x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 4.6 M 2024-01-24T14:15:53.830 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-osd x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 18 M 2024-01-24T14:15:53.830 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-prometheus-alerts noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 15 k 2024-01-24T14:15:53.830 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-selinux x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 24 k 2024-01-24T14:15:53.830 INFO:teuthology.orchestra.run.smithi123.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-01-24T14:15:53.831 INFO:teuthology.orchestra.run.smithi123.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-01-24T14:15:53.831 INFO:teuthology.orchestra.run.smithi123.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-01-24T14:15:53.831 INFO:teuthology.orchestra.run.smithi123.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2024-01-24T14:15:53.831 INFO:teuthology.orchestra.run.smithi123.stdout: krb5-devel x86_64 1.18.2-26.el8_9 rhel-8-for-x86_64-baseos-rpms 562 k 2024-01-24T14:15:53.831 INFO:teuthology.orchestra.run.smithi123.stdout: libcephsqlite x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 141 k 2024-01-24T14:15:53.831 INFO:teuthology.orchestra.run.smithi123.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2024-01-24T14:15:53.831 INFO:teuthology.orchestra.run.smithi123.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-01-24T14:15:53.831 INFO:teuthology.orchestra.run.smithi123.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-01-24T14:15:53.831 INFO:teuthology.orchestra.run.smithi123.stdout: libpq x86_64 13.11-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2024-01-24T14:15:53.831 INFO:teuthology.orchestra.run.smithi123.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-01-24T14:15:53.831 INFO:teuthology.orchestra.run.smithi123.stdout: librabbitmq x86_64 0.9.0-4.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2024-01-24T14:15:53.831 INFO:teuthology.orchestra.run.smithi123.stdout: libradosstriper1 x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 474 k 2024-01-24T14:15:53.831 INFO:teuthology.orchestra.run.smithi123.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-01-24T14:15:53.831 INFO:teuthology.orchestra.run.smithi123.stdout: librgw2 x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 6.5 M 2024-01-24T14:15:53.832 INFO:teuthology.orchestra.run.smithi123.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2024-01-24T14:15:53.832 INFO:teuthology.orchestra.run.smithi123.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2024-01-24T14:15:53.832 INFO:teuthology.orchestra.run.smithi123.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-01-24T14:15:53.832 INFO:teuthology.orchestra.run.smithi123.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2024-01-24T14:15:53.832 INFO:teuthology.orchestra.run.smithi123.stdout: libzstd-devel x86_64 1.4.4-1.el8 rhel-8-for-x86_64-baseos-rpms 44 k 2024-01-24T14:15:53.832 INFO:teuthology.orchestra.run.smithi123.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-01-24T14:15:53.832 INFO:teuthology.orchestra.run.smithi123.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2024-01-24T14:15:53.832 INFO:teuthology.orchestra.run.smithi123.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-01-24T14:15:53.832 INFO:teuthology.orchestra.run.smithi123.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-01-24T14:15:53.832 INFO:teuthology.orchestra.run.smithi123.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2024-01-24T14:15:53.832 INFO:teuthology.orchestra.run.smithi123.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2024-01-24T14:15:53.832 INFO:teuthology.orchestra.run.smithi123.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-01-24T14:15:53.833 INFO:teuthology.orchestra.run.smithi123.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-01-24T14:15:53.833 INFO:teuthology.orchestra.run.smithi123.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-01-24T14:15:53.833 INFO:teuthology.orchestra.run.smithi123.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-01-24T14:15:53.833 INFO:teuthology.orchestra.run.smithi123.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-01-24T14:15:53.833 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-01-24T14:15:53.833 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-argparse x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 45 k 2024-01-24T14:15:53.833 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-common x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 123 k 2024-01-24T14:15:53.833 INFO:teuthology.orchestra.run.smithi123.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-01-24T14:15:53.833 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-01-24T14:15:53.833 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-01-24T14:15:53.833 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-01-24T14:15:53.833 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cryptography x86_64 3.2.1-6.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2024-01-24T14:15:53.833 INFO:teuthology.orchestra.run.smithi123.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-01-24T14:15:53.834 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-01-24T14:15:53.834 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-01-24T14:15:53.834 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-01-24T14:15:53.834 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-01-24T14:15:53.834 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-01-24T14:15:53.834 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-01-24T14:15:53.834 INFO:teuthology.orchestra.run.smithi123.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-01-24T14:15:53.834 INFO:teuthology.orchestra.run.smithi123.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-01-24T14:15:53.834 INFO:teuthology.orchestra.run.smithi123.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-01-24T14:15:53.834 INFO:teuthology.orchestra.run.smithi123.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-01-24T14:15:53.834 INFO:teuthology.orchestra.run.smithi123.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-01-24T14:15:53.834 INFO:teuthology.orchestra.run.smithi123.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-01-24T14:15:53.834 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-01-24T14:15:53.834 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-01-24T14:15:53.835 INFO:teuthology.orchestra.run.smithi123.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-01-24T14:15:53.835 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-01-24T14:15:53.835 INFO:teuthology.orchestra.run.smithi123.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-01-24T14:15:53.835 INFO:teuthology.orchestra.run.smithi123.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-01-24T14:15:53.835 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-01-24T14:15:53.835 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-01-24T14:15:53.835 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-01-24T14:15:53.835 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-01-24T14:15:53.835 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-01-24T14:15:53.835 INFO:teuthology.orchestra.run.smithi123.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-01-24T14:15:53.835 INFO:teuthology.orchestra.run.smithi123.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-01-24T14:15:53.835 INFO:teuthology.orchestra.run.smithi123.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-01-24T14:15:53.835 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-01-24T14:15:53.836 INFO:teuthology.orchestra.run.smithi123.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-01-24T14:15:53.836 INFO:teuthology.orchestra.run.smithi123.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-01-24T14:15:53.836 INFO:teuthology.orchestra.run.smithi123.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-01-24T14:15:53.836 INFO:teuthology.orchestra.run.smithi123.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-01-24T14:15:53.836 INFO:teuthology.orchestra.run.smithi123.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-01-24T14:15:53.836 INFO:teuthology.orchestra.run.smithi123.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-01-24T14:15:53.836 INFO:teuthology.orchestra.run.smithi123.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-01-24T14:15:53.836 INFO:teuthology.orchestra.run.smithi123.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-01-24T14:15:53.836 INFO:teuthology.orchestra.run.smithi123.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-01-24T14:15:53.836 INFO:teuthology.orchestra.run.smithi123.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-01-24T14:15:53.836 INFO:teuthology.orchestra.run.smithi123.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-01-24T14:15:53.836 INFO:teuthology.orchestra.run.smithi123.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-01-24T14:15:53.836 INFO:teuthology.orchestra.run.smithi123.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-01-24T14:15:53.836 INFO:teuthology.orchestra.run.smithi123.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2024-01-24T14:15:53.837 INFO:teuthology.orchestra.run.smithi123.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-01-24T14:15:53.837 INFO:teuthology.orchestra.run.smithi123.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2024-01-24T14:15:53.837 INFO:teuthology.orchestra.run.smithi123.stdout:Enabling module streams: 2024-01-24T14:15:53.837 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql 10 2024-01-24T14:15:53.837 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-24T14:15:53.837 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-01-24T14:15:53.837 INFO:teuthology.orchestra.run.smithi123.stdout:=========================================================================================================================== 2024-01-24T14:15:53.837 INFO:teuthology.orchestra.run.smithi123.stdout:Install 129 Packages 2024-01-24T14:15:53.837 INFO:teuthology.orchestra.run.smithi123.stdout:Upgrade 16 Packages 2024-01-24T14:15:53.837 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-24T14:15:53.838 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 212 M 2024-01-24T14:15:53.838 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-01-24T14:15:53.972 INFO:teuthology.orchestra.run.smithi194.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-01-24T14:15:53.973 INFO:teuthology.orchestra.run.smithi194.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-01-24T14:15:53.992 INFO:teuthology.orchestra.run.smithi194.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2024-01-24T14:15:53.993 INFO:teuthology.orchestra.run.smithi194.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2024-01-24T14:15:53.995 INFO:teuthology.orchestra.run.smithi194.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2024-01-24T14:15:53.996 INFO:teuthology.orchestra.run.smithi194.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2024-01-24T14:15:53.999 INFO:teuthology.orchestra.run.smithi194.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2024-01-24T14:15:54.003 INFO:teuthology.orchestra.run.smithi194.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2024-01-24T14:15:54.005 INFO:teuthology.orchestra.run.smithi194.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2024-01-24T14:15:54.019 INFO:teuthology.orchestra.run.smithi123.stdout:(1/145): ceph-18.2.1-341.g4e3182a6.el8.x86_64.r 52 kB/s | 6.4 kB 00:00 2024-01-24T14:15:54.202 INFO:teuthology.orchestra.run.smithi123.stdout:(2/145): ceph-fuse-18.2.1-341.g4e3182a6.el8.x86 4.6 MB/s | 852 kB 00:00 2024-01-24T14:15:54.210 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-24T14:15:54.224 INFO:teuthology.orchestra.run.smithi194.stdout:=========================================================================================================================== 2024-01-24T14:15:54.224 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-01-24T14:15:54.224 INFO:teuthology.orchestra.run.smithi194.stdout:=========================================================================================================================== 2024-01-24T14:15:54.224 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-01-24T14:15:54.224 INFO:teuthology.orchestra.run.smithi194.stdout: NetworkManager-initscripts-updown noarch 1:1.40.16-13.el8_9 rhel-8-for-x86_64-baseos-rpms 144 k 2024-01-24T14:15:54.224 INFO:teuthology.orchestra.run.smithi194.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2024-01-24T14:15:54.224 INFO:teuthology.orchestra.run.smithi194.stdout: ceph x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 6.4 k 2024-01-24T14:15:54.224 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-base x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 5.4 M 2024-01-24T14:15:54.224 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-fuse x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 852 k 2024-01-24T14:15:54.225 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-immutable-object-cache x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 165 k 2024-01-24T14:15:54.225 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 1.6 M 2024-01-24T14:15:54.225 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-cephadm noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 132 k 2024-01-24T14:15:54.225 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-dashboard noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 1.8 M 2024-01-24T14:15:54.225 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 7.4 M 2024-01-24T14:15:54.225 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-rook noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 50 k 2024-01-24T14:15:54.225 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-radosgw x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 13 M 2024-01-24T14:15:54.225 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-test x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 47 M 2024-01-24T14:15:54.225 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-volume noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 262 k 2024-01-24T14:15:54.225 INFO:teuthology.orchestra.run.smithi194.stdout: cephadm noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 171 k 2024-01-24T14:15:54.225 INFO:teuthology.orchestra.run.smithi194.stdout: cephfs-mirror x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 259 k 2024-01-24T14:15:54.225 INFO:teuthology.orchestra.run.smithi194.stdout: cephfs-top noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 18 k 2024-01-24T14:15:54.226 INFO:teuthology.orchestra.run.smithi194.stdout: elfutils-libelf-devel x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 62 k 2024-01-24T14:15:54.226 INFO:teuthology.orchestra.run.smithi194.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2024-01-24T14:15:54.226 INFO:teuthology.orchestra.run.smithi194.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2024-01-24T14:15:54.226 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs-devel x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 31 k 2024-01-24T14:15:54.226 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs2 x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 776 k 2024-01-24T14:15:54.226 INFO:teuthology.orchestra.run.smithi194.stdout: librados-devel x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 139 k 2024-01-24T14:15:54.226 INFO:teuthology.orchestra.run.smithi194.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2024-01-24T14:15:54.226 INFO:teuthology.orchestra.run.smithi194.stdout: libuuid-devel x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 100 k 2024-01-24T14:15:54.226 INFO:teuthology.orchestra.run.smithi194.stdout: openssl-devel x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 2.3 M 2024-01-24T14:15:54.226 INFO:teuthology.orchestra.run.smithi194.stdout: postgresql x86_64 10.23-3.module+el8.9.0+20831+78489779 rhel-8-for-x86_64-appstream-rpms 1.5 M 2024-01-24T14:15:54.226 INFO:teuthology.orchestra.run.smithi194.stdout: postgresql-contrib x86_64 10.23-3.module+el8.9.0+20831+78489779 rhel-8-for-x86_64-appstream-rpms 811 k 2024-01-24T14:15:54.226 INFO:teuthology.orchestra.run.smithi194.stdout: postgresql-server x86_64 10.23-3.module+el8.9.0+20831+78489779 rhel-8-for-x86_64-appstream-rpms 5.1 M 2024-01-24T14:15:54.226 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cephfs x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 197 k 2024-01-24T14:15:54.226 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rados x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 371 k 2024-01-24T14:15:54.227 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rbd x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 365 k 2024-01-24T14:15:54.227 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rgw x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 107 k 2024-01-24T14:15:54.227 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-fuse x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 94 k 2024-01-24T14:15:54.227 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-mirror x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 3.4 M 2024-01-24T14:15:54.227 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-nbd x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 181 k 2024-01-24T14:15:54.227 INFO:teuthology.orchestra.run.smithi194.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 73 k 2024-01-24T14:15:54.227 INFO:teuthology.orchestra.run.smithi194.stdout:Upgrading: 2024-01-24T14:15:54.227 INFO:teuthology.orchestra.run.smithi194.stdout: elfutils-libelf x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 232 k 2024-01-24T14:15:54.227 INFO:teuthology.orchestra.run.smithi194.stdout: elfutils-libs x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 303 k 2024-01-24T14:15:54.227 INFO:teuthology.orchestra.run.smithi194.stdout: iproute x86_64 6.2.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 854 k 2024-01-24T14:15:54.227 INFO:teuthology.orchestra.run.smithi194.stdout: libblkid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-01-24T14:15:54.227 INFO:teuthology.orchestra.run.smithi194.stdout: libfdisk x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 254 k 2024-01-24T14:15:54.227 INFO:teuthology.orchestra.run.smithi194.stdout: libmount x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 237 k 2024-01-24T14:15:54.227 INFO:teuthology.orchestra.run.smithi194.stdout: librados2 x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 3.7 M 2024-01-24T14:15:54.227 INFO:teuthology.orchestra.run.smithi194.stdout: librbd1 x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 3.6 M 2024-01-24T14:15:54.228 INFO:teuthology.orchestra.run.smithi194.stdout: libsmartcols x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 179 k 2024-01-24T14:15:54.228 INFO:teuthology.orchestra.run.smithi194.stdout: libuuid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 99 k 2024-01-24T14:15:54.228 INFO:teuthology.orchestra.run.smithi194.stdout: openssl x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 711 k 2024-01-24T14:15:54.228 INFO:teuthology.orchestra.run.smithi194.stdout: openssl-libs x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 1.5 M 2024-01-24T14:15:54.228 INFO:teuthology.orchestra.run.smithi194.stdout: util-linux x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 2.5 M 2024-01-24T14:15:54.228 INFO:teuthology.orchestra.run.smithi194.stdout: util-linux-user x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 103 k 2024-01-24T14:15:54.228 INFO:teuthology.orchestra.run.smithi194.stdout: xfsdump x86_64 3.1.8-7.el8_9 rhel-8-for-x86_64-baseos-rpms 333 k 2024-01-24T14:15:54.228 INFO:teuthology.orchestra.run.smithi194.stdout: xfsprogs x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M 2024-01-24T14:15:54.228 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-01-24T14:15:54.228 INFO:teuthology.orchestra.run.smithi194.stdout: automake noarch 1.16.1-8.el8 rhel-8-for-x86_64-appstream-rpms 714 k 2024-01-24T14:15:54.228 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-common x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 22 M 2024-01-24T14:15:54.228 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-grafana-dashboards noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 23 k 2024-01-24T14:15:54.228 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mds x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 2.3 M 2024-01-24T14:15:54.228 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-modules-core noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 242 k 2024-01-24T14:15:54.229 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mon x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 4.6 M 2024-01-24T14:15:54.229 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-osd x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 18 M 2024-01-24T14:15:54.229 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-prometheus-alerts noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 15 k 2024-01-24T14:15:54.229 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-selinux x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 24 k 2024-01-24T14:15:54.229 INFO:teuthology.orchestra.run.smithi194.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-01-24T14:15:54.229 INFO:teuthology.orchestra.run.smithi194.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-01-24T14:15:54.229 INFO:teuthology.orchestra.run.smithi194.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-01-24T14:15:54.229 INFO:teuthology.orchestra.run.smithi194.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2024-01-24T14:15:54.229 INFO:teuthology.orchestra.run.smithi194.stdout: krb5-devel x86_64 1.18.2-26.el8_9 rhel-8-for-x86_64-baseos-rpms 562 k 2024-01-24T14:15:54.229 INFO:teuthology.orchestra.run.smithi194.stdout: libcephsqlite x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 141 k 2024-01-24T14:15:54.229 INFO:teuthology.orchestra.run.smithi194.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2024-01-24T14:15:54.229 INFO:teuthology.orchestra.run.smithi194.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-01-24T14:15:54.229 INFO:teuthology.orchestra.run.smithi194.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-01-24T14:15:54.230 INFO:teuthology.orchestra.run.smithi194.stdout: libpq x86_64 13.11-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2024-01-24T14:15:54.230 INFO:teuthology.orchestra.run.smithi194.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-01-24T14:15:54.230 INFO:teuthology.orchestra.run.smithi194.stdout: librabbitmq x86_64 0.9.0-4.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2024-01-24T14:15:54.230 INFO:teuthology.orchestra.run.smithi194.stdout: libradosstriper1 x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 474 k 2024-01-24T14:15:54.230 INFO:teuthology.orchestra.run.smithi194.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-01-24T14:15:54.230 INFO:teuthology.orchestra.run.smithi194.stdout: librgw2 x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 6.5 M 2024-01-24T14:15:54.230 INFO:teuthology.orchestra.run.smithi194.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2024-01-24T14:15:54.230 INFO:teuthology.orchestra.run.smithi194.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2024-01-24T14:15:54.230 INFO:teuthology.orchestra.run.smithi194.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-01-24T14:15:54.230 INFO:teuthology.orchestra.run.smithi194.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2024-01-24T14:15:54.230 INFO:teuthology.orchestra.run.smithi194.stdout: libzstd-devel x86_64 1.4.4-1.el8 rhel-8-for-x86_64-baseos-rpms 44 k 2024-01-24T14:15:54.230 INFO:teuthology.orchestra.run.smithi194.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-01-24T14:15:54.230 INFO:teuthology.orchestra.run.smithi194.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2024-01-24T14:15:54.231 INFO:teuthology.orchestra.run.smithi194.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-01-24T14:15:54.231 INFO:teuthology.orchestra.run.smithi194.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-01-24T14:15:54.231 INFO:teuthology.orchestra.run.smithi194.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2024-01-24T14:15:54.231 INFO:teuthology.orchestra.run.smithi194.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2024-01-24T14:15:54.231 INFO:teuthology.orchestra.run.smithi194.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-01-24T14:15:54.231 INFO:teuthology.orchestra.run.smithi194.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-01-24T14:15:54.231 INFO:teuthology.orchestra.run.smithi194.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-01-24T14:15:54.231 INFO:teuthology.orchestra.run.smithi194.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-01-24T14:15:54.231 INFO:teuthology.orchestra.run.smithi194.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-01-24T14:15:54.231 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-01-24T14:15:54.231 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-argparse x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 45 k 2024-01-24T14:15:54.231 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-common x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 123 k 2024-01-24T14:15:54.231 INFO:teuthology.orchestra.run.smithi194.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-01-24T14:15:54.231 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-01-24T14:15:54.231 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-01-24T14:15:54.232 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-01-24T14:15:54.232 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cryptography x86_64 3.2.1-6.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2024-01-24T14:15:54.232 INFO:teuthology.orchestra.run.smithi194.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-01-24T14:15:54.232 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-01-24T14:15:54.232 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-01-24T14:15:54.232 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-01-24T14:15:54.232 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-01-24T14:15:54.232 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-01-24T14:15:54.232 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-01-24T14:15:54.232 INFO:teuthology.orchestra.run.smithi194.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-01-24T14:15:54.232 INFO:teuthology.orchestra.run.smithi194.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-01-24T14:15:54.232 INFO:teuthology.orchestra.run.smithi194.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-01-24T14:15:54.232 INFO:teuthology.orchestra.run.smithi194.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-01-24T14:15:54.233 INFO:teuthology.orchestra.run.smithi194.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-01-24T14:15:54.233 INFO:teuthology.orchestra.run.smithi194.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-01-24T14:15:54.233 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-01-24T14:15:54.233 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-01-24T14:15:54.233 INFO:teuthology.orchestra.run.smithi194.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-01-24T14:15:54.233 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-01-24T14:15:54.233 INFO:teuthology.orchestra.run.smithi194.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-01-24T14:15:54.233 INFO:teuthology.orchestra.run.smithi194.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-01-24T14:15:54.233 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-01-24T14:15:54.233 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-01-24T14:15:54.233 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-01-24T14:15:54.233 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-01-24T14:15:54.233 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-01-24T14:15:54.233 INFO:teuthology.orchestra.run.smithi194.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-01-24T14:15:54.233 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-01-24T14:15:54.234 INFO:teuthology.orchestra.run.smithi194.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-01-24T14:15:54.234 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-01-24T14:15:54.234 INFO:teuthology.orchestra.run.smithi194.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-01-24T14:15:54.234 INFO:teuthology.orchestra.run.smithi194.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-01-24T14:15:54.234 INFO:teuthology.orchestra.run.smithi194.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-01-24T14:15:54.234 INFO:teuthology.orchestra.run.smithi194.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-01-24T14:15:54.234 INFO:teuthology.orchestra.run.smithi194.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-01-24T14:15:54.234 INFO:teuthology.orchestra.run.smithi194.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-01-24T14:15:54.234 INFO:teuthology.orchestra.run.smithi194.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-01-24T14:15:54.234 INFO:teuthology.orchestra.run.smithi194.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-01-24T14:15:54.234 INFO:teuthology.orchestra.run.smithi194.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-01-24T14:15:54.234 INFO:teuthology.orchestra.run.smithi194.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-01-24T14:15:54.234 INFO:teuthology.orchestra.run.smithi194.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-01-24T14:15:54.234 INFO:teuthology.orchestra.run.smithi194.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-01-24T14:15:54.234 INFO:teuthology.orchestra.run.smithi194.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-01-24T14:15:54.235 INFO:teuthology.orchestra.run.smithi194.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2024-01-24T14:15:54.235 INFO:teuthology.orchestra.run.smithi194.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-01-24T14:15:54.235 INFO:teuthology.orchestra.run.smithi194.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2024-01-24T14:15:54.235 INFO:teuthology.orchestra.run.smithi194.stdout:Enabling module streams: 2024-01-24T14:15:54.235 INFO:teuthology.orchestra.run.smithi194.stdout: postgresql 10 2024-01-24T14:15:54.235 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-24T14:15:54.235 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-01-24T14:15:54.235 INFO:teuthology.orchestra.run.smithi194.stdout:=========================================================================================================================== 2024-01-24T14:15:54.235 INFO:teuthology.orchestra.run.smithi194.stdout:Install 129 Packages 2024-01-24T14:15:54.235 INFO:teuthology.orchestra.run.smithi194.stdout:Upgrade 16 Packages 2024-01-24T14:15:54.235 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-24T14:15:54.236 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 212 M 2024-01-24T14:15:54.236 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-01-24T14:15:54.278 INFO:teuthology.orchestra.run.smithi123.stdout:(3/145): ceph-immutable-object-cache-18.2.1-341 2.1 MB/s | 165 kB 00:00 2024-01-24T14:15:54.514 INFO:teuthology.orchestra.run.smithi194.stdout:(1/145): ceph-18.2.1-341.g4e3182a6.el8.x86_64.r 30 kB/s | 6.4 kB 00:00 2024-01-24T14:15:54.532 INFO:teuthology.orchestra.run.smithi123.stdout:(4/145): ceph-mds-18.2.1-341.g4e3182a6.el8.x86_ 9.0 MB/s | 2.3 MB 00:00 2024-01-24T14:15:54.995 INFO:teuthology.orchestra.run.smithi123.stdout:(5/145): ceph-mgr-18.2.1-341.g4e3182a6.el8.x86_ 3.4 MB/s | 1.6 MB 00:00 2024-01-24T14:15:55.071 INFO:teuthology.orchestra.run.smithi123.stdout:(6/145): ceph-base-18.2.1-341.g4e3182a6.el8.x86 4.6 MB/s | 5.4 MB 00:01 2024-01-24T14:15:55.100 INFO:teuthology.orchestra.run.smithi194.stdout:(2/145): ceph-fuse-18.2.1-341.g4e3182a6.el8.x86 1.4 MB/s | 852 kB 00:00 2024-01-24T14:15:55.167 INFO:teuthology.orchestra.run.smithi194.stdout:(3/145): ceph-base-18.2.1-341.g4e3182a6.el8.x86 6.2 MB/s | 5.4 MB 00:00 2024-01-24T14:15:55.267 INFO:teuthology.orchestra.run.smithi194.stdout:(4/145): ceph-immutable-object-cache-18.2.1-341 993 kB/s | 165 kB 00:00 2024-01-24T14:15:55.734 INFO:teuthology.orchestra.run.smithi194.stdout:(5/145): ceph-mds-18.2.1-341.g4e3182a6.el8.x86_ 4.0 MB/s | 2.3 MB 00:00 2024-01-24T14:15:55.739 INFO:teuthology.orchestra.run.smithi123.stdout:(7/145): ceph-mon-18.2.1-341.g4e3182a6.el8.x86_ 6.2 MB/s | 4.6 MB 00:00 2024-01-24T14:15:56.067 INFO:teuthology.orchestra.run.smithi194.stdout:(6/145): ceph-mgr-18.2.1-341.g4e3182a6.el8.x86_ 1.9 MB/s | 1.6 MB 00:00 2024-01-24T14:15:56.809 INFO:teuthology.orchestra.run.smithi194.stdout:(7/145): ceph-mon-18.2.1-341.g4e3182a6.el8.x86_ 4.3 MB/s | 4.6 MB 00:01 2024-01-24T14:15:57.481 INFO:teuthology.orchestra.run.smithi123.stdout:(8/145): ceph-common-18.2.1-341.g4e3182a6.el8.x 6.1 MB/s | 22 MB 00:03 2024-01-24T14:15:57.573 INFO:teuthology.orchestra.run.smithi123.stdout:(9/145): ceph-selinux-18.2.1-341.g4e3182a6.el8. 266 kB/s | 24 kB 00:00 2024-01-24T14:15:57.740 INFO:teuthology.orchestra.run.smithi123.stdout:(10/145): ceph-radosgw-18.2.1-341.g4e3182a6.el8 6.5 MB/s | 13 MB 00:01 2024-01-24T14:15:57.840 INFO:teuthology.orchestra.run.smithi123.stdout:(11/145): cephfs-mirror-18.2.1-341.g4e3182a6.el 2.5 MB/s | 259 kB 00:00 2024-01-24T14:15:57.924 INFO:teuthology.orchestra.run.smithi123.stdout:(12/145): libcephfs-devel-18.2.1-341.g4e3182a6. 371 kB/s | 31 kB 00:00 2024-01-24T14:15:58.091 INFO:teuthology.orchestra.run.smithi123.stdout:(13/145): ceph-osd-18.2.1-341.g4e3182a6.el8.x86 5.9 MB/s | 18 MB 00:03 2024-01-24T14:15:58.147 INFO:teuthology.orchestra.run.smithi123.stdout:(14/145): libcephfs2-18.2.1-341.g4e3182a6.el8.x 3.4 MB/s | 776 kB 00:00 2024-01-24T14:15:58.183 INFO:teuthology.orchestra.run.smithi123.stdout:(15/145): libcephsqlite-18.2.1-341.g4e3182a6.el 1.5 MB/s | 141 kB 00:00 2024-01-24T14:15:58.208 INFO:teuthology.orchestra.run.smithi123.stdout:(16/145): librados-devel-18.2.1-341.g4e3182a6.e 2.2 MB/s | 139 kB 00:00 2024-01-24T14:15:58.350 INFO:teuthology.orchestra.run.smithi123.stdout:(17/145): libradosstriper1-18.2.1-341.g4e3182a6 2.8 MB/s | 474 kB 00:00 2024-01-24T14:15:58.450 INFO:teuthology.orchestra.run.smithi123.stdout:(18/145): python3-ceph-argparse-18.2.1-341.g4e3 447 kB/s | 45 kB 00:00 2024-01-24T14:15:58.550 INFO:teuthology.orchestra.run.smithi123.stdout:(19/145): python3-ceph-common-18.2.1-341.g4e318 1.2 MB/s | 123 kB 00:00 2024-01-24T14:15:58.651 INFO:teuthology.orchestra.run.smithi123.stdout:(20/145): python3-cephfs-18.2.1-341.g4e3182a6.e 1.9 MB/s | 197 kB 00:00 2024-01-24T14:15:58.776 INFO:teuthology.orchestra.run.smithi123.stdout:(21/145): python3-rados-18.2.1-341.g4e3182a6.el 2.9 MB/s | 371 kB 00:00 2024-01-24T14:15:58.893 INFO:teuthology.orchestra.run.smithi123.stdout:(22/145): python3-rbd-18.2.1-341.g4e3182a6.el8. 3.1 MB/s | 365 kB 00:00 2024-01-24T14:15:59.002 INFO:teuthology.orchestra.run.smithi123.stdout:(23/145): python3-rgw-18.2.1-341.g4e3182a6.el8. 985 kB/s | 107 kB 00:00 2024-01-24T14:15:59.102 INFO:teuthology.orchestra.run.smithi123.stdout:(24/145): rbd-fuse-18.2.1-341.g4e3182a6.el8.x86 944 kB/s | 94 kB 00:00 2024-01-24T14:15:59.302 INFO:teuthology.orchestra.run.smithi123.stdout:(25/145): librgw2-18.2.1-341.g4e3182a6.el8.x86_ 6.0 MB/s | 6.5 MB 00:01 2024-01-24T14:15:59.386 INFO:teuthology.orchestra.run.smithi123.stdout:(26/145): rbd-nbd-18.2.1-341.g4e3182a6.el8.x86_ 2.1 MB/s | 181 kB 00:00 2024-01-24T14:15:59.444 INFO:teuthology.orchestra.run.smithi123.stdout:(27/145): ceph-grafana-dashboards-18.2.1-341.g4 396 kB/s | 23 kB 00:00 2024-01-24T14:15:59.459 INFO:teuthology.orchestra.run.smithi194.stdout:(8/145): ceph-radosgw-18.2.1-341.g4e3182a6.el8. 4.9 MB/s | 13 MB 00:02 2024-01-24T14:15:59.526 INFO:teuthology.orchestra.run.smithi194.stdout:(9/145): ceph-selinux-18.2.1-341.g4e3182a6.el8. 365 kB/s | 24 kB 00:00 2024-01-24T14:15:59.528 INFO:teuthology.orchestra.run.smithi123.stdout:(28/145): ceph-mgr-cephadm-18.2.1-341.g4e3182a6 1.5 MB/s | 132 kB 00:00 2024-01-24T14:15:59.795 INFO:teuthology.orchestra.run.smithi123.stdout:(29/145): rbd-mirror-18.2.1-341.g4e3182a6.el8.x 5.0 MB/s | 3.4 MB 00:00 2024-01-24T14:15:59.837 INFO:teuthology.orchestra.run.smithi123.stdout:(30/145): ceph-mgr-dashboard-18.2.1-341.g4e3182 5.7 MB/s | 1.8 MB 00:00 2024-01-24T14:15:59.843 INFO:teuthology.orchestra.run.smithi194.stdout:(10/145): ceph-osd-18.2.1-341.g4e3182a6.el8.x86 4.7 MB/s | 18 MB 00:03 2024-01-24T14:15:59.903 INFO:teuthology.orchestra.run.smithi123.stdout:(31/145): ceph-mgr-modules-core-18.2.1-341.g4e3 3.6 MB/s | 242 kB 00:00 2024-01-24T14:15:59.943 INFO:teuthology.orchestra.run.smithi194.stdout:(11/145): cephfs-mirror-18.2.1-341.g4e3182a6.el 2.5 MB/s | 259 kB 00:00 2024-01-24T14:15:59.979 INFO:teuthology.orchestra.run.smithi123.stdout:(32/145): ceph-mgr-rook-18.2.1-341.g4e3182a6.el 673 kB/s | 50 kB 00:00 2024-01-24T14:15:59.993 INFO:teuthology.orchestra.run.smithi194.stdout:(12/145): libcephfs-devel-18.2.1-341.g4e3182a6. 618 kB/s | 31 kB 00:00 2024-01-24T14:16:00.071 INFO:teuthology.orchestra.run.smithi123.stdout:(33/145): ceph-prometheus-alerts-18.2.1-341.g4e 159 kB/s | 15 kB 00:00 2024-01-24T14:16:00.154 INFO:teuthology.orchestra.run.smithi123.stdout:(34/145): ceph-volume-18.2.1-341.g4e3182a6.el8. 3.1 MB/s | 262 kB 00:00 2024-01-24T14:16:00.185 INFO:teuthology.orchestra.run.smithi194.stdout:(13/145): libcephfs2-18.2.1-341.g4e3182a6.el8.x 4.0 MB/s | 776 kB 00:00 2024-01-24T14:16:00.221 INFO:teuthology.orchestra.run.smithi123.stdout:(35/145): cephadm-18.2.1-341.g4e3182a6.el8.noar 2.5 MB/s | 171 kB 00:00 2024-01-24T14:16:00.260 INFO:teuthology.orchestra.run.smithi194.stdout:(14/145): libcephsqlite-18.2.1-341.g4e3182a6.el 1.8 MB/s | 141 kB 00:00 2024-01-24T14:16:00.288 INFO:teuthology.orchestra.run.smithi123.stdout:(36/145): cephfs-top-18.2.1-341.g4e3182a6.el8.n 271 kB/s | 18 kB 00:00 2024-01-24T14:16:00.319 INFO:teuthology.orchestra.run.smithi194.stdout:(15/145): librados-devel-18.2.1-341.g4e3182a6.e 2.3 MB/s | 139 kB 00:00 2024-01-24T14:16:00.453 INFO:teuthology.orchestra.run.smithi194.stdout:(16/145): libradosstriper1-18.2.1-341.g4e3182a6 3.5 MB/s | 474 kB 00:00 2024-01-24T14:16:00.501 INFO:teuthology.orchestra.run.smithi019.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-01-24T14:16:00.502 INFO:teuthology.orchestra.run.smithi019.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-01-24T14:16:00.523 INFO:teuthology.orchestra.run.smithi019.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2024-01-24T14:16:00.524 INFO:teuthology.orchestra.run.smithi019.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2024-01-24T14:16:00.525 INFO:teuthology.orchestra.run.smithi019.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2024-01-24T14:16:00.527 INFO:teuthology.orchestra.run.smithi019.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2024-01-24T14:16:00.530 INFO:teuthology.orchestra.run.smithi019.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2024-01-24T14:16:00.535 INFO:teuthology.orchestra.run.smithi019.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2024-01-24T14:16:00.536 INFO:teuthology.orchestra.run.smithi019.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2024-01-24T14:16:00.778 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-01-24T14:16:00.794 INFO:teuthology.orchestra.run.smithi019.stdout:=========================================================================================================================== 2024-01-24T14:16:00.794 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-01-24T14:16:00.794 INFO:teuthology.orchestra.run.smithi019.stdout:=========================================================================================================================== 2024-01-24T14:16:00.794 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-01-24T14:16:00.794 INFO:teuthology.orchestra.run.smithi019.stdout: NetworkManager-initscripts-updown noarch 1:1.40.16-13.el8_9 rhel-8-for-x86_64-baseos-rpms 144 k 2024-01-24T14:16:00.795 INFO:teuthology.orchestra.run.smithi019.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2024-01-24T14:16:00.795 INFO:teuthology.orchestra.run.smithi019.stdout: ceph x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 6.4 k 2024-01-24T14:16:00.795 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-base x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 5.4 M 2024-01-24T14:16:00.795 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-fuse x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 852 k 2024-01-24T14:16:00.795 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-immutable-object-cache x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 165 k 2024-01-24T14:16:00.795 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 1.6 M 2024-01-24T14:16:00.795 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-cephadm noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 132 k 2024-01-24T14:16:00.795 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-dashboard noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 1.8 M 2024-01-24T14:16:00.795 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 7.4 M 2024-01-24T14:16:00.795 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-rook noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 50 k 2024-01-24T14:16:00.795 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-radosgw x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 13 M 2024-01-24T14:16:00.795 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-test x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 47 M 2024-01-24T14:16:00.795 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-volume noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 262 k 2024-01-24T14:16:00.796 INFO:teuthology.orchestra.run.smithi019.stdout: cephadm noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 171 k 2024-01-24T14:16:00.796 INFO:teuthology.orchestra.run.smithi019.stdout: cephfs-mirror x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 259 k 2024-01-24T14:16:00.796 INFO:teuthology.orchestra.run.smithi019.stdout: cephfs-top noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 18 k 2024-01-24T14:16:00.796 INFO:teuthology.orchestra.run.smithi019.stdout: elfutils-libelf-devel x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 62 k 2024-01-24T14:16:00.796 INFO:teuthology.orchestra.run.smithi019.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2024-01-24T14:16:00.796 INFO:teuthology.orchestra.run.smithi019.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2024-01-24T14:16:00.796 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs-devel x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 31 k 2024-01-24T14:16:00.796 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs2 x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 776 k 2024-01-24T14:16:00.796 INFO:teuthology.orchestra.run.smithi019.stdout: librados-devel x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 139 k 2024-01-24T14:16:00.796 INFO:teuthology.orchestra.run.smithi019.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2024-01-24T14:16:00.796 INFO:teuthology.orchestra.run.smithi019.stdout: libuuid-devel x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 100 k 2024-01-24T14:16:00.796 INFO:teuthology.orchestra.run.smithi019.stdout: openssl-devel x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 2.3 M 2024-01-24T14:16:00.797 INFO:teuthology.orchestra.run.smithi019.stdout: postgresql x86_64 10.23-3.module+el8.9.0+20831+78489779 rhel-8-for-x86_64-appstream-rpms 1.5 M 2024-01-24T14:16:00.797 INFO:teuthology.orchestra.run.smithi019.stdout: postgresql-contrib x86_64 10.23-3.module+el8.9.0+20831+78489779 rhel-8-for-x86_64-appstream-rpms 811 k 2024-01-24T14:16:00.797 INFO:teuthology.orchestra.run.smithi019.stdout: postgresql-server x86_64 10.23-3.module+el8.9.0+20831+78489779 rhel-8-for-x86_64-appstream-rpms 5.1 M 2024-01-24T14:16:00.797 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cephfs x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 197 k 2024-01-24T14:16:00.797 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rados x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 371 k 2024-01-24T14:16:00.797 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rbd x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 365 k 2024-01-24T14:16:00.797 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rgw x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 107 k 2024-01-24T14:16:00.797 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-fuse x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 94 k 2024-01-24T14:16:00.797 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-mirror x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 3.4 M 2024-01-24T14:16:00.797 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-nbd x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 181 k 2024-01-24T14:16:00.797 INFO:teuthology.orchestra.run.smithi019.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 73 k 2024-01-24T14:16:00.797 INFO:teuthology.orchestra.run.smithi019.stdout:Upgrading: 2024-01-24T14:16:00.797 INFO:teuthology.orchestra.run.smithi019.stdout: elfutils-libelf x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 232 k 2024-01-24T14:16:00.798 INFO:teuthology.orchestra.run.smithi019.stdout: elfutils-libs x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 303 k 2024-01-24T14:16:00.798 INFO:teuthology.orchestra.run.smithi019.stdout: iproute x86_64 6.2.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 854 k 2024-01-24T14:16:00.798 INFO:teuthology.orchestra.run.smithi019.stdout: libblkid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-01-24T14:16:00.798 INFO:teuthology.orchestra.run.smithi019.stdout: libfdisk x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 254 k 2024-01-24T14:16:00.798 INFO:teuthology.orchestra.run.smithi019.stdout: libmount x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 237 k 2024-01-24T14:16:00.798 INFO:teuthology.orchestra.run.smithi019.stdout: librados2 x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 3.7 M 2024-01-24T14:16:00.798 INFO:teuthology.orchestra.run.smithi019.stdout: librbd1 x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 3.6 M 2024-01-24T14:16:00.798 INFO:teuthology.orchestra.run.smithi019.stdout: libsmartcols x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 179 k 2024-01-24T14:16:00.798 INFO:teuthology.orchestra.run.smithi019.stdout: libuuid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 99 k 2024-01-24T14:16:00.798 INFO:teuthology.orchestra.run.smithi019.stdout: openssl x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 711 k 2024-01-24T14:16:00.798 INFO:teuthology.orchestra.run.smithi019.stdout: openssl-libs x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 1.5 M 2024-01-24T14:16:00.798 INFO:teuthology.orchestra.run.smithi019.stdout: util-linux x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 2.5 M 2024-01-24T14:16:00.798 INFO:teuthology.orchestra.run.smithi019.stdout: util-linux-user x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 103 k 2024-01-24T14:16:00.799 INFO:teuthology.orchestra.run.smithi019.stdout: xfsdump x86_64 3.1.8-7.el8_9 rhel-8-for-x86_64-baseos-rpms 333 k 2024-01-24T14:16:00.799 INFO:teuthology.orchestra.run.smithi019.stdout: xfsprogs x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M 2024-01-24T14:16:00.799 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-01-24T14:16:00.799 INFO:teuthology.orchestra.run.smithi019.stdout: automake noarch 1.16.1-8.el8 rhel-8-for-x86_64-appstream-rpms 714 k 2024-01-24T14:16:00.799 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-common x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 22 M 2024-01-24T14:16:00.799 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-grafana-dashboards noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 23 k 2024-01-24T14:16:00.799 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mds x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 2.3 M 2024-01-24T14:16:00.799 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-modules-core noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 242 k 2024-01-24T14:16:00.799 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mon x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 4.6 M 2024-01-24T14:16:00.799 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-osd x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 18 M 2024-01-24T14:16:00.799 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-prometheus-alerts noarch 2:18.2.1-341.g4e3182a6.el8 ceph-noarch 15 k 2024-01-24T14:16:00.799 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-selinux x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 24 k 2024-01-24T14:16:00.799 INFO:teuthology.orchestra.run.smithi019.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-01-24T14:16:00.800 INFO:teuthology.orchestra.run.smithi019.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-01-24T14:16:00.800 INFO:teuthology.orchestra.run.smithi019.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-01-24T14:16:00.800 INFO:teuthology.orchestra.run.smithi019.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2024-01-24T14:16:00.800 INFO:teuthology.orchestra.run.smithi019.stdout: krb5-devel x86_64 1.18.2-26.el8_9 rhel-8-for-x86_64-baseos-rpms 562 k 2024-01-24T14:16:00.800 INFO:teuthology.orchestra.run.smithi019.stdout: libcephsqlite x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 141 k 2024-01-24T14:16:00.800 INFO:teuthology.orchestra.run.smithi019.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2024-01-24T14:16:00.800 INFO:teuthology.orchestra.run.smithi019.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-01-24T14:16:00.800 INFO:teuthology.orchestra.run.smithi019.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-01-24T14:16:00.800 INFO:teuthology.orchestra.run.smithi019.stdout: libpq x86_64 13.11-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2024-01-24T14:16:00.800 INFO:teuthology.orchestra.run.smithi019.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-01-24T14:16:00.800 INFO:teuthology.orchestra.run.smithi019.stdout: librabbitmq x86_64 0.9.0-4.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2024-01-24T14:16:00.800 INFO:teuthology.orchestra.run.smithi019.stdout: libradosstriper1 x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 474 k 2024-01-24T14:16:00.800 INFO:teuthology.orchestra.run.smithi019.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-01-24T14:16:00.801 INFO:teuthology.orchestra.run.smithi019.stdout: librgw2 x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 6.5 M 2024-01-24T14:16:00.801 INFO:teuthology.orchestra.run.smithi019.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2024-01-24T14:16:00.801 INFO:teuthology.orchestra.run.smithi019.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2024-01-24T14:16:00.801 INFO:teuthology.orchestra.run.smithi019.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-01-24T14:16:00.801 INFO:teuthology.orchestra.run.smithi019.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2024-01-24T14:16:00.801 INFO:teuthology.orchestra.run.smithi019.stdout: libzstd-devel x86_64 1.4.4-1.el8 rhel-8-for-x86_64-baseos-rpms 44 k 2024-01-24T14:16:00.801 INFO:teuthology.orchestra.run.smithi019.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-01-24T14:16:00.801 INFO:teuthology.orchestra.run.smithi019.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2024-01-24T14:16:00.801 INFO:teuthology.orchestra.run.smithi019.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-01-24T14:16:00.801 INFO:teuthology.orchestra.run.smithi019.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-01-24T14:16:00.801 INFO:teuthology.orchestra.run.smithi019.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2024-01-24T14:16:00.801 INFO:teuthology.orchestra.run.smithi019.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2024-01-24T14:16:00.801 INFO:teuthology.orchestra.run.smithi019.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-01-24T14:16:00.802 INFO:teuthology.orchestra.run.smithi019.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-01-24T14:16:00.802 INFO:teuthology.orchestra.run.smithi019.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-01-24T14:16:00.802 INFO:teuthology.orchestra.run.smithi019.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-01-24T14:16:00.802 INFO:teuthology.orchestra.run.smithi019.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-01-24T14:16:00.802 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-01-24T14:16:00.802 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-argparse x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 45 k 2024-01-24T14:16:00.802 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-common x86_64 2:18.2.1-341.g4e3182a6.el8 ceph 123 k 2024-01-24T14:16:00.802 INFO:teuthology.orchestra.run.smithi019.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-01-24T14:16:00.802 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-01-24T14:16:00.802 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-01-24T14:16:00.802 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-01-24T14:16:00.802 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cryptography x86_64 3.2.1-6.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2024-01-24T14:16:00.802 INFO:teuthology.orchestra.run.smithi019.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-01-24T14:16:00.803 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-01-24T14:16:00.803 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-01-24T14:16:00.803 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-01-24T14:16:00.803 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-01-24T14:16:00.803 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-01-24T14:16:00.803 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-01-24T14:16:00.803 INFO:teuthology.orchestra.run.smithi019.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-01-24T14:16:00.803 INFO:teuthology.orchestra.run.smithi019.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-01-24T14:16:00.803 INFO:teuthology.orchestra.run.smithi019.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-01-24T14:16:00.803 INFO:teuthology.orchestra.run.smithi019.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-01-24T14:16:00.803 INFO:teuthology.orchestra.run.smithi019.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-01-24T14:16:00.803 INFO:teuthology.orchestra.run.smithi019.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-01-24T14:16:00.803 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-01-24T14:16:00.804 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-01-24T14:16:00.804 INFO:teuthology.orchestra.run.smithi019.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-01-24T14:16:00.804 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-01-24T14:16:00.804 INFO:teuthology.orchestra.run.smithi019.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-01-24T14:16:00.804 INFO:teuthology.orchestra.run.smithi019.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-01-24T14:16:00.804 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-01-24T14:16:00.804 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-01-24T14:16:00.804 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-01-24T14:16:00.804 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-01-24T14:16:00.804 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-01-24T14:16:00.804 INFO:teuthology.orchestra.run.smithi019.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-01-24T14:16:00.804 INFO:teuthology.orchestra.run.smithi019.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-01-24T14:16:00.804 INFO:teuthology.orchestra.run.smithi019.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-01-24T14:16:00.805 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-01-24T14:16:00.805 INFO:teuthology.orchestra.run.smithi019.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-01-24T14:16:00.805 INFO:teuthology.orchestra.run.smithi019.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-01-24T14:16:00.805 INFO:teuthology.orchestra.run.smithi019.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-01-24T14:16:00.805 INFO:teuthology.orchestra.run.smithi019.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-01-24T14:16:00.805 INFO:teuthology.orchestra.run.smithi019.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-01-24T14:16:00.805 INFO:teuthology.orchestra.run.smithi019.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-01-24T14:16:00.805 INFO:teuthology.orchestra.run.smithi019.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-01-24T14:16:00.805 INFO:teuthology.orchestra.run.smithi019.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-01-24T14:16:00.805 INFO:teuthology.orchestra.run.smithi019.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-01-24T14:16:00.805 INFO:teuthology.orchestra.run.smithi019.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-01-24T14:16:00.805 INFO:teuthology.orchestra.run.smithi019.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-01-24T14:16:00.805 INFO:teuthology.orchestra.run.smithi019.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-01-24T14:16:00.805 INFO:teuthology.orchestra.run.smithi019.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-01-24T14:16:00.805 INFO:teuthology.orchestra.run.smithi019.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2024-01-24T14:16:00.806 INFO:teuthology.orchestra.run.smithi019.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-01-24T14:16:00.806 INFO:teuthology.orchestra.run.smithi019.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2024-01-24T14:16:00.806 INFO:teuthology.orchestra.run.smithi019.stdout:Enabling module streams: 2024-01-24T14:16:00.806 INFO:teuthology.orchestra.run.smithi019.stdout: postgresql 10 2024-01-24T14:16:00.806 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:16:00.806 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-01-24T14:16:00.806 INFO:teuthology.orchestra.run.smithi019.stdout:=========================================================================================================================== 2024-01-24T14:16:00.806 INFO:teuthology.orchestra.run.smithi019.stdout:Install 129 Packages 2024-01-24T14:16:00.806 INFO:teuthology.orchestra.run.smithi019.stdout:Upgrade 16 Packages 2024-01-24T14:16:00.806 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:16:00.807 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 212 M 2024-01-24T14:16:00.807 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-01-24T14:16:01.006 INFO:teuthology.orchestra.run.smithi123.stdout:(37/145): ceph-mgr-diskprediction-local-18.2.1- 6.1 MB/s | 7.4 MB 00:01 2024-01-24T14:16:01.053 INFO:teuthology.orchestra.run.smithi194.stdout:(17/145): ceph-common-18.2.1-341.g4e3182a6.el8. 3.2 MB/s | 22 MB 00:06 2024-01-24T14:16:01.129 INFO:teuthology.orchestra.run.smithi194.stdout:(18/145): python3-ceph-argparse-18.2.1-341.g4e3 596 kB/s | 45 kB 00:00 2024-01-24T14:16:01.203 INFO:teuthology.orchestra.run.smithi194.stdout:(19/145): python3-ceph-common-18.2.1-341.g4e318 1.6 MB/s | 123 kB 00:00 2024-01-24T14:16:01.298 INFO:teuthology.orchestra.run.smithi123.stdout:(38/145): gperftools-libs-2.7-9.el8.x86_64.rpm 303 kB/s | 306 kB 00:01 2024-01-24T14:16:01.304 INFO:teuthology.orchestra.run.smithi194.stdout:(20/145): python3-cephfs-18.2.1-341.g4e3182a6.e 1.9 MB/s | 197 kB 00:00 2024-01-24T14:16:01.349 INFO:teuthology.orchestra.run.smithi019.stdout:(1/145): ceph-18.2.1-341.g4e3182a6.el8.x86_64.r 34 kB/s | 6.4 kB 00:00 2024-01-24T14:16:01.431 INFO:teuthology.orchestra.run.smithi123.stdout:(39/145): libunwind-1.3.1-3.el8.x86_64.rpm 563 kB/s | 75 kB 00:00 2024-01-24T14:16:01.446 INFO:teuthology.orchestra.run.smithi194.stdout:(21/145): python3-rados-18.2.1-341.g4e3182a6.el 2.6 MB/s | 371 kB 00:00 2024-01-24T14:16:01.521 INFO:teuthology.orchestra.run.smithi194.stdout:(22/145): librgw2-18.2.1-341.g4e3182a6.el8.x86_ 6.1 MB/s | 6.5 MB 00:01 2024-01-24T14:16:01.588 INFO:teuthology.orchestra.run.smithi194.stdout:(23/145): python3-rgw-18.2.1-341.g4e3182a6.el8. 1.6 MB/s | 107 kB 00:00 2024-01-24T14:16:01.613 INFO:teuthology.orchestra.run.smithi194.stdout:(24/145): python3-rbd-18.2.1-341.g4e3182a6.el8. 2.1 MB/s | 365 kB 00:00 2024-01-24T14:16:01.672 INFO:teuthology.orchestra.run.smithi194.stdout:(25/145): rbd-fuse-18.2.1-341.g4e3182a6.el8.x86 1.1 MB/s | 94 kB 00:00 2024-01-24T14:16:01.687 INFO:teuthology.orchestra.run.smithi123.stdout:(40/145): python3-asyncssh-2.7.0-2.el8.noarch.r 1.8 MB/s | 476 kB 00:00 2024-01-24T14:16:01.740 INFO:teuthology.orchestra.run.smithi123.stdout:(41/145): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 824 kB/s | 44 kB 00:00 2024-01-24T14:16:01.747 INFO:teuthology.orchestra.run.smithi194.stdout:(26/145): rbd-nbd-18.2.1-341.g4e3182a6.el8.x86_ 2.4 MB/s | 181 kB 00:00 2024-01-24T14:16:01.814 INFO:teuthology.orchestra.run.smithi194.stdout:(27/145): ceph-grafana-dashboards-18.2.1-341.g4 347 kB/s | 23 kB 00:00 2024-01-24T14:16:01.824 INFO:teuthology.orchestra.run.smithi123.stdout:(42/145): python3-beautifulsoup4-4.6.3-2.el8.1. 2.2 MB/s | 185 kB 00:00 2024-01-24T14:16:01.824 INFO:teuthology.orchestra.run.smithi019.stdout:(2/145): ceph-fuse-18.2.1-341.g4e3182a6.el8.x86 1.8 MB/s | 852 kB 00:00 2024-01-24T14:16:01.866 INFO:teuthology.orchestra.run.smithi123.stdout:(43/145): python3-cachetools-3.1.1-4.el8.noarch 794 kB/s | 33 kB 00:00 2024-01-24T14:16:01.872 INFO:teuthology.orchestra.run.smithi194.stdout:(28/145): ceph-mgr-cephadm-18.2.1-341.g4e3182a6 2.2 MB/s | 132 kB 00:00 2024-01-24T14:16:01.916 INFO:teuthology.orchestra.run.smithi123.stdout:(44/145): python3-certifi-2018.10.15-7.el8.noar 327 kB/s | 16 kB 00:00 2024-01-24T14:16:01.942 INFO:teuthology.orchestra.run.smithi019.stdout:(3/145): ceph-immutable-object-cache-18.2.1-341 1.4 MB/s | 165 kB 00:00 2024-01-24T14:16:02.016 INFO:teuthology.orchestra.run.smithi123.stdout:(45/145): python3-cheroot-8.5.2-1.el8.noarch.rp 1.7 MB/s | 173 kB 00:00 2024-01-24T14:16:02.125 INFO:teuthology.orchestra.run.smithi123.stdout:(46/145): python3-cherrypy-18.4.0-1.el8.noarch. 3.5 MB/s | 384 kB 00:00 2024-01-24T14:16:02.175 INFO:teuthology.orchestra.run.smithi123.stdout:(47/145): python3-google-auth-1.1.1-10.el8.noar 1.6 MB/s | 81 kB 00:00 2024-01-24T14:16:02.239 INFO:teuthology.orchestra.run.smithi194.stdout:(29/145): ceph-mgr-dashboard-18.2.1-341.g4e3182 4.8 MB/s | 1.8 MB 00:00 2024-01-24T14:16:02.242 INFO:teuthology.orchestra.run.smithi123.stdout:(48/145): liboath-2.6.2-3.el8.x86_64.rpm 48 kB/s | 59 kB 00:01 2024-01-24T14:16:02.267 INFO:teuthology.orchestra.run.smithi123.stdout:(49/145): python3-jaraco-6.2-6.el8.noarch.rpm 121 kB/s | 11 kB 00:00 2024-01-24T14:16:02.292 INFO:teuthology.orchestra.run.smithi123.stdout:(50/145): python3-jaraco-functools-2.0-4.el8.no 350 kB/s | 18 kB 00:00 2024-01-24T14:16:02.351 INFO:teuthology.orchestra.run.smithi123.stdout:(51/145): python3-logutils-0.3.5-11.el8.noarch. 838 kB/s | 49 kB 00:00 2024-01-24T14:16:02.418 INFO:teuthology.orchestra.run.smithi123.stdout:(52/145): python3-more-itertools-7.2.0-3.el8.no 880 kB/s | 59 kB 00:00 2024-01-24T14:16:02.431 INFO:teuthology.orchestra.run.smithi194.stdout:(30/145): rbd-mirror-18.2.1-341.g4e3182a6.el8.x 4.2 MB/s | 3.4 MB 00:00 2024-01-24T14:16:02.510 INFO:teuthology.orchestra.run.smithi123.stdout:(53/145): python3-natsort-7.1.1-2.el8.noarch.rp 658 kB/s | 60 kB 00:00 2024-01-24T14:16:02.531 INFO:teuthology.orchestra.run.smithi194.stdout:(31/145): ceph-mgr-modules-core-18.2.1-341.g4e3 2.4 MB/s | 242 kB 00:00 2024-01-24T14:16:02.535 INFO:teuthology.orchestra.run.smithi123.stdout:(54/145): python3-kubernetes-11.0.0-6.el8.noarc 4.4 MB/s | 1.2 MB 00:00 2024-01-24T14:16:02.585 INFO:teuthology.orchestra.run.smithi123.stdout:(55/145): python3-portend-2.6-1.el8.noarch.rpm 313 kB/s | 16 kB 00:00 2024-01-24T14:16:02.611 INFO:teuthology.orchestra.run.smithi123.stdout:(56/145): python3-pecan-1.3.2-9.el8.noarch.rpm 2.7 MB/s | 283 kB 00:00 2024-01-24T14:16:02.641 INFO:teuthology.orchestra.run.smithi194.stdout:(32/145): ceph-mgr-rook-18.2.1-341.g4e3182a6.el 463 kB/s | 50 kB 00:00 2024-01-24T14:16:02.653 INFO:teuthology.orchestra.run.smithi123.stdout:(57/145): python3-repoze-lru-0.7-6.el8.noarch.r 503 kB/s | 34 kB 00:00 2024-01-24T14:16:02.704 INFO:teuthology.orchestra.run.smithi123.stdout:(58/145): python3-routes-2.4.1-12.el8.noarch.rp 2.1 MB/s | 196 kB 00:00 2024-01-24T14:16:02.728 INFO:teuthology.orchestra.run.smithi123.stdout:(59/145): python3-rsa-4.9-2.el8.noarch.rpm 855 kB/s | 64 kB 00:00 2024-01-24T14:16:02.756 INFO:teuthology.orchestra.run.smithi194.stdout:(33/145): ceph-prometheus-alerts-18.2.1-341.g4e 126 kB/s | 15 kB 00:00 2024-01-24T14:16:02.775 INFO:teuthology.orchestra.run.smithi019.stdout:(4/145): ceph-mds-18.2.1-341.g4e3182a6.el8.x86_ 2.8 MB/s | 2.3 MB 00:00 2024-01-24T14:16:02.795 INFO:teuthology.orchestra.run.smithi123.stdout:(60/145): python3-simplegeneric-0.8.1-17.el8.no 204 kB/s | 19 kB 00:00 2024-01-24T14:16:02.821 INFO:teuthology.orchestra.run.smithi123.stdout:(61/145): python3-singledispatch-3.4.0.3-18.el8 260 kB/s | 24 kB 00:00 2024-01-24T14:16:02.857 INFO:teuthology.orchestra.run.smithi194.stdout:(34/145): ceph-volume-18.2.1-341.g4e3182a6.el8. 2.5 MB/s | 262 kB 00:00 2024-01-24T14:16:02.862 INFO:teuthology.orchestra.run.smithi123.stdout:(62/145): python3-tempora-1.14.1-5.el8.noarch.r 438 kB/s | 29 kB 00:00 2024-01-24T14:16:02.888 INFO:teuthology.orchestra.run.smithi123.stdout:(63/145): python3-trustme-0.6.0-4.el8.noarch.rp 412 kB/s | 27 kB 00:00 2024-01-24T14:16:02.955 INFO:teuthology.orchestra.run.smithi123.stdout:(64/145): python3-webob-1.8.5-1.el8.1.noarch.rp 3.7 MB/s | 251 kB 00:00 2024-01-24T14:16:02.958 INFO:teuthology.orchestra.run.smithi194.stdout:(35/145): cephadm-18.2.1-341.g4e3182a6.el8.noar 1.7 MB/s | 171 kB 00:00 2024-01-24T14:16:02.980 INFO:teuthology.orchestra.run.smithi123.stdout:(65/145): python3-waitress-1.4.3-1.el8.noarch.r 2.1 MB/s | 255 kB 00:00 2024-01-24T14:16:03.050 INFO:teuthology.orchestra.run.smithi194.stdout:(36/145): cephfs-top-18.2.1-341.g4e3182a6.el8.n 196 kB/s | 18 kB 00:00 2024-01-24T14:16:03.053 INFO:teuthology.orchestra.run.smithi123.stdout:(66/145): python3-websocket-client-0.56.0-5.el8 627 kB/s | 61 kB 00:00 2024-01-24T14:16:03.097 INFO:teuthology.orchestra.run.smithi123.stdout:(67/145): python3-webtest-2.0.33-1.el8.noarch.r 729 kB/s | 85 kB 00:00 2024-01-24T14:16:03.123 INFO:teuthology.orchestra.run.smithi123.stdout:(68/145): python3-zc-lockfile-2.0-2.el8.noarch. 330 kB/s | 23 kB 00:00 2024-01-24T14:16:03.148 INFO:teuthology.orchestra.run.smithi123.stdout:(69/145): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.8 MB/s | 70 kB 00:00 2024-01-24T14:16:03.275 INFO:teuthology.orchestra.run.smithi019.stdout:(5/145): ceph-base-18.2.1-341.g4e3182a6.el8.x86 2.6 MB/s | 5.4 MB 00:02 2024-01-24T14:16:03.309 INFO:teuthology.orchestra.run.smithi019.stdout:(6/145): ceph-mgr-18.2.1-341.g4e3182a6.el8.x86_ 2.9 MB/s | 1.6 MB 00:00 2024-01-24T14:16:03.348 INFO:teuthology.orchestra.run.smithi123.stdout:(70/145): libverto-devel-0.3.0-5.el8.x86_64.rpm 89 kB/s | 18 kB 00:00 2024-01-24T14:16:03.532 INFO:teuthology.orchestra.run.smithi123.stdout:(71/145): thrift-0.13.0-2.el8.x86_64.rpm 4.0 MB/s | 1.7 MB 00:00 2024-01-24T14:16:03.557 INFO:teuthology.orchestra.run.smithi123.stdout:(72/145): python3-jwt-1.6.1-2.el8.noarch.rpm 204 kB/s | 43 kB 00:00 2024-01-24T14:16:03.600 INFO:teuthology.orchestra.run.smithi194.stdout:(37/145): gperftools-libs-2.7-9.el8.x86_64.rpm 557 kB/s | 306 kB 00:00 2024-01-24T14:16:03.676 INFO:teuthology.orchestra.run.smithi194.stdout:(38/145): ceph-mgr-diskprediction-local-18.2.1- 5.2 MB/s | 7.4 MB 00:01 2024-01-24T14:16:03.709 INFO:teuthology.orchestra.run.smithi194.stdout:(39/145): liboath-2.6.2-3.el8.x86_64.rpm 542 kB/s | 59 kB 00:00 2024-01-24T14:16:03.758 INFO:teuthology.orchestra.run.smithi123.stdout:(73/145): python3-pycparser-2.14-14.el8.noarch. 561 kB/s | 109 kB 00:00 2024-01-24T14:16:03.791 INFO:teuthology.orchestra.run.smithi123.stdout:(74/145): python3-oauthlib-2.1.0-1.el8.noarch.r 598 kB/s | 155 kB 00:00 2024-01-24T14:16:03.892 INFO:teuthology.orchestra.run.smithi194.stdout:(40/145): python3-asyncssh-2.7.0-2.el8.noarch.r 2.5 MB/s | 476 kB 00:00 2024-01-24T14:16:04.001 INFO:teuthology.orchestra.run.smithi194.stdout:(41/145): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 402 kB/s | 44 kB 00:00 2024-01-24T14:16:04.026 INFO:teuthology.orchestra.run.smithi194.stdout:(42/145): libunwind-1.3.1-3.el8.x86_64.rpm 214 kB/s | 75 kB 00:00 2024-01-24T14:16:04.101 INFO:teuthology.orchestra.run.smithi194.stdout:(43/145): python3-beautifulsoup4-4.6.3-2.el8.1. 1.8 MB/s | 185 kB 00:00 2024-01-24T14:16:04.109 INFO:teuthology.orchestra.run.smithi019.stdout:(7/145): ceph-mon-18.2.1-341.g4e3182a6.el8.x86_ 5.5 MB/s | 4.6 MB 00:00 2024-01-24T14:16:04.119 INFO:teuthology.orchestra.run.smithi194.stdout:(44/145): python3-cachetools-3.1.1-4.el8.noarch 359 kB/s | 33 kB 00:00 2024-01-24T14:16:04.185 INFO:teuthology.orchestra.run.smithi194.stdout:(45/145): python3-certifi-2018.10.15-7.el8.noar 195 kB/s | 16 kB 00:00 2024-01-24T14:16:04.208 INFO:teuthology.orchestra.run.smithi123.stdout:(75/145): ceph-test-18.2.1-341.g4e3182a6.el8.x8 7.0 MB/s | 47 MB 00:06 2024-01-24T14:16:04.233 INFO:teuthology.orchestra.run.smithi123.stdout:(76/145): python3-requests-oauthlib-1.0.0-1.el8 90 kB/s | 43 kB 00:00 2024-01-24T14:16:04.267 INFO:teuthology.orchestra.run.smithi123.stdout:(77/145): libaio-devel-0.3.112-1.el8.x86_64.rpm 39 kB/s | 19 kB 00:00 2024-01-24T14:16:04.294 INFO:teuthology.orchestra.run.smithi194.stdout:(46/145): python3-cherrypy-18.4.0-1.el8.noarch. 3.5 MB/s | 384 kB 00:00 2024-01-24T14:16:04.319 INFO:teuthology.orchestra.run.smithi194.stdout:(47/145): python3-cheroot-8.5.2-1.el8.noarch.rp 860 kB/s | 173 kB 00:00 2024-01-24T14:16:04.403 INFO:teuthology.orchestra.run.smithi194.stdout:(48/145): python3-google-auth-1.1.1-10.el8.noar 750 kB/s | 81 kB 00:00 2024-01-24T14:16:04.420 INFO:teuthology.orchestra.run.smithi194.stdout:(49/145): python3-jaraco-6.2-6.el8.noarch.rpm 111 kB/s | 11 kB 00:00 2024-01-24T14:16:04.484 INFO:teuthology.orchestra.run.smithi123.stdout:(78/145): libzstd-devel-1.4.4-1.el8.x86_64.rpm 159 kB/s | 44 kB 00:00 2024-01-24T14:16:04.512 INFO:teuthology.orchestra.run.smithi194.stdout:(50/145): python3-jaraco-functools-2.0-4.el8.no 162 kB/s | 18 kB 00:00 2024-01-24T14:16:04.620 INFO:teuthology.orchestra.run.smithi194.stdout:(51/145): python3-logutils-0.3.5-11.el8.noarch. 450 kB/s | 49 kB 00:00 2024-01-24T14:16:04.625 INFO:teuthology.orchestra.run.smithi123.stdout:(79/145): pcre2-devel-10.32-2.el8.x86_64.rpm 1.5 MB/s | 605 kB 00:00 2024-01-24T14:16:04.684 INFO:teuthology.orchestra.run.smithi123.stdout:(80/145): pcre2-utf16-10.32-2.el8.x86_64.rpm 549 kB/s | 229 kB 00:00 2024-01-24T14:16:04.729 INFO:teuthology.orchestra.run.smithi194.stdout:(52/145): python3-more-itertools-7.2.0-3.el8.no 542 kB/s | 59 kB 00:00 2024-01-24T14:16:04.768 INFO:teuthology.orchestra.run.smithi123.stdout:(81/145): pcre2-utf32-10.32-2.el8.x86_64.rpm 777 kB/s | 220 kB 00:00 2024-01-24T14:16:04.826 INFO:teuthology.orchestra.run.smithi123.stdout:(82/145): libselinux-devel-2.9-5.el8.x86_64.rpm 997 kB/s | 200 kB 00:00 2024-01-24T14:16:04.837 INFO:teuthology.orchestra.run.smithi194.stdout:(53/145): python3-natsort-7.1.1-2.el8.noarch.rp 556 kB/s | 60 kB 00:00 2024-01-24T14:16:04.880 INFO:teuthology.orchestra.run.smithi194.stdout:(54/145): python3-kubernetes-11.0.0-6.el8.noarc 2.5 MB/s | 1.2 MB 00:00 2024-01-24T14:16:04.902 INFO:teuthology.orchestra.run.smithi123.stdout:(83/145): keyutils-libs-devel-1.5.10-9.el8.x86_ 221 kB/s | 48 kB 00:00 2024-01-24T14:16:04.971 INFO:teuthology.orchestra.run.smithi194.stdout:(55/145): python3-pecan-1.3.2-9.el8.noarch.rpm 2.1 MB/s | 283 kB 00:00 2024-01-24T14:16:04.977 INFO:teuthology.orchestra.run.smithi123.stdout:(84/145): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 384 kB/s | 58 kB 00:00 2024-01-24T14:16:04.996 INFO:teuthology.orchestra.run.smithi194.stdout:(56/145): python3-portend-2.6-1.el8.noarch.rpm 133 kB/s | 16 kB 00:00 2024-01-24T14:16:05.036 INFO:teuthology.orchestra.run.smithi123.stdout:(85/145): libcom_err-devel-1.45.6-4.el8.x86_64. 289 kB/s | 39 kB 00:00 2024-01-24T14:16:05.061 INFO:teuthology.orchestra.run.smithi123.stdout:(86/145): libsepol-devel-2.9-3.el8.x86_64.rpm 297 kB/s | 87 kB 00:00 2024-01-24T14:16:05.063 INFO:teuthology.orchestra.run.smithi194.stdout:(57/145): python3-repoze-lru-0.7-6.el8.noarch.r 366 kB/s | 34 kB 00:00 2024-01-24T14:16:05.097 INFO:teuthology.orchestra.run.smithi194.stdout:(58/145): python3-routes-2.4.1-12.el8.noarch.rp 1.9 MB/s | 196 kB 00:00 2024-01-24T14:16:05.156 INFO:teuthology.orchestra.run.smithi194.stdout:(59/145): python3-rsa-4.9-2.el8.noarch.rpm 698 kB/s | 64 kB 00:00 2024-01-24T14:16:05.206 INFO:teuthology.orchestra.run.smithi194.stdout:(60/145): python3-simplegeneric-0.8.1-17.el8.no 173 kB/s | 19 kB 00:00 2024-01-24T14:16:05.211 INFO:teuthology.orchestra.run.smithi123.stdout:(87/145): fuse-2.9.7-17.el8.x86_64.rpm 355 kB/s | 83 kB 00:00 2024-01-24T14:16:05.244 INFO:teuthology.orchestra.run.smithi123.stdout:(88/145): libgfortran-8.5.0-20.el8.x86_64.rpm 3.0 MB/s | 645 kB 00:00 2024-01-24T14:16:05.248 INFO:teuthology.orchestra.run.smithi194.stdout:(61/145): python3-singledispatch-3.4.0.3-18.el8 259 kB/s | 24 kB 00:00 2024-01-24T14:16:05.312 INFO:teuthology.orchestra.run.smithi123.stdout:(89/145): libuuid-devel-2.32.1-43.el8.x86_64.rp 994 kB/s | 100 kB 00:00 2024-01-24T14:16:05.335 INFO:teuthology.orchestra.run.smithi194.stdout:(62/145): python3-tempora-1.14.1-5.el8.noarch.r 227 kB/s | 29 kB 00:00 2024-01-24T14:16:05.337 INFO:teuthology.orchestra.run.smithi123.stdout:(90/145): libquadmath-8.5.0-20.el8.x86_64.rpm 625 kB/s | 172 kB 00:00 2024-01-24T14:16:05.365 INFO:teuthology.orchestra.run.smithi194.stdout:(63/145): python3-trustme-0.6.0-4.el8.noarch.rp 235 kB/s | 27 kB 00:00 2024-01-24T14:16:05.370 INFO:teuthology.orchestra.run.smithi123.stdout:(91/145): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.9 MB/s | 238 kB 00:00 2024-01-24T14:16:05.418 INFO:teuthology.orchestra.run.smithi019.stdout:(8/145): ceph-common-18.2.1-341.g4e3182a6.el8.x 5.1 MB/s | 22 MB 00:04 2024-01-24T14:16:05.432 INFO:teuthology.orchestra.run.smithi194.stdout:(64/145): python3-waitress-1.4.3-1.el8.noarch.r 2.6 MB/s | 255 kB 00:00 2024-01-24T14:16:05.437 INFO:teuthology.orchestra.run.smithi123.stdout:(92/145): python3-cryptography-3.2.1-6.el8.x86_ 4.4 MB/s | 559 kB 00:00 2024-01-24T14:16:05.466 INFO:teuthology.orchestra.run.smithi194.stdout:(65/145): python3-webob-1.8.5-1.el8.1.noarch.rp 2.4 MB/s | 251 kB 00:00 2024-01-24T14:16:05.533 INFO:teuthology.orchestra.run.smithi194.stdout:(66/145): python3-websocket-client-0.56.0-5.el8 611 kB/s | 61 kB 00:00 2024-01-24T14:16:05.558 INFO:teuthology.orchestra.run.smithi194.stdout:(67/145): python3-webtest-2.0.33-1.el8.noarch.r 928 kB/s | 85 kB 00:00 2024-01-24T14:16:05.559 INFO:teuthology.orchestra.run.smithi019.stdout:(9/145): ceph-selinux-18.2.1-341.g4e3182a6.el8. 172 kB/s | 24 kB 00:00 2024-01-24T14:16:05.738 INFO:teuthology.orchestra.run.smithi123.stdout:(93/145): xfsprogs-devel-5.0.0-12.el8.x86_64.rp 181 kB/s | 73 kB 00:00 2024-01-24T14:16:05.763 INFO:teuthology.orchestra.run.smithi123.stdout:(94/145): librabbitmq-0.9.0-4.el8.x86_64.rpm 144 kB/s | 47 kB 00:00 2024-01-24T14:16:05.797 INFO:teuthology.orchestra.run.smithi123.stdout:(95/145): elfutils-libelf-devel-0.189-3.el8.x86 145 kB/s | 62 kB 00:00 2024-01-24T14:16:05.810 INFO:teuthology.orchestra.run.smithi019.stdout:(10/145): ceph-osd-18.2.1-341.g4e3182a6.el8.x86 7.1 MB/s | 18 MB 00:02 2024-01-24T14:16:05.842 INFO:teuthology.orchestra.run.smithi194.stdout:(68/145): python3-zc-lockfile-2.0-2.el8.noarch. 74 kB/s | 23 kB 00:00 2024-01-24T14:16:05.939 INFO:teuthology.orchestra.run.smithi123.stdout:(96/145): krb5-devel-1.18.2-26.el8_9.x86_64.rpm 2.7 MB/s | 562 kB 00:00 2024-01-24T14:16:05.943 INFO:teuthology.orchestra.run.smithi019.stdout:(11/145): ceph-radosgw-18.2.1-341.g4e3182a6.el8 7.1 MB/s | 13 MB 00:01 2024-01-24T14:16:05.969 INFO:teuthology.orchestra.run.smithi019.stdout:(12/145): cephfs-mirror-18.2.1-341.g4e3182a6.el 1.6 MB/s | 259 kB 00:00 2024-01-24T14:16:05.984 INFO:teuthology.orchestra.run.smithi194.stdout:(69/145): xmlstarlet-1.6.1-20.el8.x86_64.rpm 492 kB/s | 70 kB 00:00 2024-01-24T14:16:05.985 INFO:teuthology.orchestra.run.smithi019.stdout:(13/145): libcephfs-devel-18.2.1-341.g4e3182a6. 747 kB/s | 31 kB 00:00 2024-01-24T14:16:06.034 INFO:teuthology.orchestra.run.smithi194.stdout:(70/145): thrift-0.13.0-2.el8.x86_64.rpm 3.6 MB/s | 1.7 MB 00:00 2024-01-24T14:16:06.044 INFO:teuthology.orchestra.run.smithi019.stdout:(14/145): libcephsqlite-18.2.1-341.g4e3182a6.el 2.4 MB/s | 141 kB 00:00 2024-01-24T14:16:06.086 INFO:teuthology.orchestra.run.smithi019.stdout:(15/145): libcephfs2-18.2.1-341.g4e3182a6.el8.x 6.5 MB/s | 776 kB 00:00 2024-01-24T14:16:06.111 INFO:teuthology.orchestra.run.smithi019.stdout:(16/145): librados-devel-18.2.1-341.g4e3182a6.e 2.0 MB/s | 139 kB 00:00 2024-01-24T14:16:06.186 INFO:teuthology.orchestra.run.smithi019.stdout:(17/145): libradosstriper1-18.2.1-341.g4e3182a6 4.6 MB/s | 474 kB 00:00 2024-01-24T14:16:06.201 INFO:teuthology.orchestra.run.smithi194.stdout:(71/145): python3-jsonpointer-1.10-11.el8.noarc 92 kB/s | 20 kB 00:00 2024-01-24T14:16:06.214 INFO:teuthology.orchestra.run.smithi123.stdout:(97/145): python3-jsonpointer-1.10-11.el8.noarc 73 kB/s | 20 kB 00:00 2024-01-24T14:16:06.231 INFO:teuthology.orchestra.run.smithi123.stdout:(98/145): NetworkManager-initscripts-updown-1.4 332 kB/s | 144 kB 00:00 2024-01-24T14:16:06.245 INFO:teuthology.orchestra.run.smithi019.stdout:(18/145): python3-ceph-argparse-18.2.1-341.g4e3 764 kB/s | 45 kB 00:00 2024-01-24T14:16:06.301 INFO:teuthology.orchestra.run.smithi194.stdout:(72/145): python3-prettytable-0.7.2-14.el8.noar 168 kB/s | 44 kB 00:00 2024-01-24T14:16:06.320 INFO:teuthology.orchestra.run.smithi019.stdout:(19/145): python3-ceph-common-18.2.1-341.g4e318 1.6 MB/s | 123 kB 00:00 2024-01-24T14:16:06.335 INFO:teuthology.orchestra.run.smithi194.stdout:(73/145): python3-pyasn1-modules-0.3.7-6.el8.no 825 kB/s | 110 kB 00:00 2024-01-24T14:16:06.339 INFO:teuthology.orchestra.run.smithi123.stdout:(99/145): openssl-devel-1.1.1k-12.el8_9.x86_64. 4.0 MB/s | 2.3 MB 00:00 2024-01-24T14:16:06.365 INFO:teuthology.orchestra.run.smithi123.stdout:(100/145): python3-prettytable-0.7.2-14.el8.noa 292 kB/s | 44 kB 00:00 2024-01-24T14:16:06.390 INFO:teuthology.orchestra.run.smithi123.stdout:(101/145): python3-pyasn1-modules-0.3.7-6.el8.n 694 kB/s | 110 kB 00:00 2024-01-24T14:16:06.404 INFO:teuthology.orchestra.run.smithi019.stdout:(20/145): python3-cephfs-18.2.1-341.g4e3182a6.e 2.3 MB/s | 197 kB 00:00 2024-01-24T14:16:06.473 INFO:teuthology.orchestra.run.smithi123.stdout:(102/145): python3-jsonpatch-1.21-2.el8.noarch. 198 kB/s | 27 kB 00:00 2024-01-24T14:16:06.477 INFO:teuthology.orchestra.run.smithi194.stdout:(74/145): python3-jsonpatch-1.21-2.el8.noarch.r 151 kB/s | 27 kB 00:00 2024-01-24T14:16:06.496 INFO:teuthology.orchestra.run.smithi019.stdout:(21/145): python3-rados-18.2.1-341.g4e3182a6.el 4.0 MB/s | 371 kB 00:00 2024-01-24T14:16:06.498 INFO:teuthology.orchestra.run.smithi123.stdout:(103/145): python3-pyasn1-0.3.7-6.el8.noarch.rp 944 kB/s | 126 kB 00:00 2024-01-24T14:16:06.527 INFO:teuthology.orchestra.run.smithi194.stdout:(75/145): python3-pyasn1-0.3.7-6.el8.noarch.rpm 659 kB/s | 126 kB 00:00 2024-01-24T14:16:06.541 INFO:teuthology.orchestra.run.smithi123.stdout:(104/145): python3-werkzeug-0.12.2-4.el8.noarch 3.0 MB/s | 457 kB 00:00 2024-01-24T14:16:06.596 INFO:teuthology.orchestra.run.smithi019.stdout:(22/145): python3-rbd-18.2.1-341.g4e3182a6.el8. 3.6 MB/s | 365 kB 00:00 2024-01-24T14:16:06.619 INFO:teuthology.orchestra.run.smithi194.stdout:(76/145): python3-werkzeug-0.12.2-4.el8.noarch. 3.1 MB/s | 457 kB 00:00 2024-01-24T14:16:06.633 INFO:teuthology.orchestra.run.smithi123.stdout:(105/145): bison-3.0.4-10.el8.x86_64.rpm 4.2 MB/s | 688 kB 00:00 2024-01-24T14:16:06.653 INFO:teuthology.orchestra.run.smithi194.stdout:(77/145): bison-3.0.4-10.el8.x86_64.rpm 5.4 MB/s | 688 kB 00:00 2024-01-24T14:16:06.658 INFO:teuthology.orchestra.run.smithi123.stdout:(106/145): flex-2.6.1-9.el8.x86_64.rpm 2.0 MB/s | 320 kB 00:00 2024-01-24T14:16:06.671 INFO:teuthology.orchestra.run.smithi019.stdout:(23/145): python3-rgw-18.2.1-341.g4e3182a6.el8. 1.4 MB/s | 107 kB 00:00 2024-01-24T14:16:06.711 INFO:teuthology.orchestra.run.smithi194.stdout:(78/145): flex-2.6.1-9.el8.x86_64.rpm 3.4 MB/s | 320 kB 00:00 2024-01-24T14:16:06.742 INFO:teuthology.orchestra.run.smithi123.stdout:(107/145): libtool-2.4.6-25.el8.x86_64.rpm 3.5 MB/s | 709 kB 00:00 2024-01-24T14:16:06.755 INFO:teuthology.orchestra.run.smithi019.stdout:(24/145): rbd-fuse-18.2.1-341.g4e3182a6.el8.x86 1.1 MB/s | 94 kB 00:00 2024-01-24T14:16:06.767 INFO:teuthology.orchestra.run.smithi123.stdout:(108/145): python3-markupsafe-0.23-19.el8.x86_6 290 kB/s | 39 kB 00:00 2024-01-24T14:16:06.795 INFO:teuthology.orchestra.run.smithi194.stdout:(79/145): python3-markupsafe-0.23-19.el8.x86_64 466 kB/s | 39 kB 00:00 2024-01-24T14:16:06.850 INFO:teuthology.orchestra.run.smithi123.stdout:(109/145): oniguruma-6.8.2-2.el8.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2024-01-24T14:16:06.853 INFO:teuthology.orchestra.run.smithi194.stdout:(80/145): libtool-2.4.6-25.el8.x86_64.rpm 3.5 MB/s | 709 kB 00:00 2024-01-24T14:16:06.875 INFO:teuthology.orchestra.run.smithi123.stdout:(110/145): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.2 MB/s | 259 kB 00:00 2024-01-24T14:16:06.895 INFO:teuthology.orchestra.run.smithi194.stdout:(81/145): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.5 MB/s | 259 kB 00:00 2024-01-24T14:16:06.901 INFO:teuthology.orchestra.run.smithi123.stdout:(111/145): uuid-1.6.2-43.el8.x86_64.rpm 478 kB/s | 64 kB 00:00 2024-01-24T14:16:06.964 INFO:teuthology.orchestra.run.smithi019.stdout:(25/145): librgw2-18.2.1-341.g4e3182a6.el8.x86_ 7.7 MB/s | 6.5 MB 00:00 2024-01-24T14:16:07.004 INFO:teuthology.orchestra.run.smithi194.stdout:(82/145): oniguruma-6.8.2-2.el8.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-01-24T14:16:07.009 INFO:teuthology.orchestra.run.smithi123.stdout:(112/145): python3-pyOpenSSL-19.0.0-1.el8.noarc 651 kB/s | 103 kB 00:00 2024-01-24T14:16:07.029 INFO:teuthology.orchestra.run.smithi194.stdout:(83/145): uuid-1.6.2-43.el8.x86_64.rpm 479 kB/s | 64 kB 00:00 2024-01-24T14:16:07.051 INFO:teuthology.orchestra.run.smithi123.stdout:(113/145): python3-jinja2-2.10.1-3.el8.noarch.r 3.5 MB/s | 538 kB 00:00 2024-01-24T14:16:07.056 INFO:teuthology.orchestra.run.smithi019.stdout:(26/145): rbd-nbd-18.2.1-341.g4e3182a6.el8.x86_ 1.9 MB/s | 181 kB 00:00 2024-01-24T14:16:07.113 INFO:teuthology.orchestra.run.smithi194.stdout:(84/145): python3-pyOpenSSL-19.0.0-1.el8.noarch 954 kB/s | 103 kB 00:00 2024-01-24T14:16:07.131 INFO:teuthology.orchestra.run.smithi019.stdout:(27/145): ceph-grafana-dashboards-18.2.1-341.g4 308 kB/s | 23 kB 00:00 2024-01-24T14:16:07.152 INFO:teuthology.orchestra.run.smithi123.stdout:(114/145): python3-babel-2.5.1-7.el8.noarch.rpm 17 MB/s | 4.8 MB 00:00 2024-01-24T14:16:07.210 INFO:teuthology.orchestra.run.smithi123.stdout:(115/145): socat-1.7.4.1-1.el8.x86_64.rpm 2.0 MB/s | 323 kB 00:00 2024-01-24T14:16:07.240 INFO:teuthology.orchestra.run.smithi019.stdout:(28/145): rbd-mirror-18.2.1-341.g4e3182a6.el8.x 7.1 MB/s | 3.4 MB 00:00 2024-01-24T14:16:07.255 INFO:teuthology.orchestra.run.smithi194.stdout:(85/145): python3-jinja2-2.10.1-3.el8.noarch.rp 3.7 MB/s | 538 kB 00:00 2024-01-24T14:16:07.265 INFO:teuthology.orchestra.run.smithi019.stdout:(29/145): ceph-mgr-cephadm-18.2.1-341.g4e3182a6 987 kB/s | 132 kB 00:00 2024-01-24T14:16:07.294 INFO:teuthology.orchestra.run.smithi123.stdout:(116/145): python3-numpy-1.14.3-10.el8.x86_64.r 13 MB/s | 3.7 MB 00:00 2024-01-24T14:16:07.319 INFO:teuthology.orchestra.run.smithi123.stdout:(117/145): python3-numpy-f2py-1.14.3-10.el8.x86 1.3 MB/s | 225 kB 00:00 2024-01-24T14:16:07.322 INFO:teuthology.orchestra.run.smithi194.stdout:(86/145): python3-babel-2.5.1-7.el8.noarch.rpm 16 MB/s | 4.8 MB 00:00 2024-01-24T14:16:07.352 INFO:teuthology.orchestra.run.smithi123.stdout:(118/145): librdkafka-0.11.4-3.el8.x86_64.rpm 2.4 MB/s | 354 kB 00:00 2024-01-24T14:16:07.447 INFO:teuthology.orchestra.run.smithi194.stdout:(87/145): python3-numpy-1.14.3-10.el8.x86_64.rp 19 MB/s | 3.7 MB 00:00 2024-01-24T14:16:07.491 INFO:teuthology.orchestra.run.smithi019.stdout:(30/145): ceph-mgr-dashboard-18.2.1-341.g4e3182 7.0 MB/s | 1.8 MB 00:00 2024-01-24T14:16:07.561 INFO:teuthology.orchestra.run.smithi123.stdout:(119/145): openblas-threads-0.3.15-6.el8.x86_64 19 MB/s | 5.2 MB 00:00 2024-01-24T14:16:07.568 INFO:teuthology.orchestra.run.smithi194.stdout:(88/145): socat-1.7.4.1-1.el8.x86_64.rpm 1.3 MB/s | 323 kB 00:00 2024-01-24T14:16:07.582 INFO:teuthology.orchestra.run.smithi019.stdout:(31/145): ceph-mgr-modules-core-18.2.1-341.g4e3 2.6 MB/s | 242 kB 00:00 2024-01-24T14:16:07.586 INFO:teuthology.orchestra.run.smithi123.stdout:(120/145): python3-mako-1.0.6-14.el8.noarch.rpm 587 kB/s | 157 kB 00:00 2024-01-24T14:16:07.597 INFO:teuthology.orchestra.run.smithi194.stdout:(89/145): python3-numpy-f2py-1.14.3-10.el8.x86_ 1.5 MB/s | 225 kB 00:00 2024-01-24T14:16:07.658 INFO:teuthology.orchestra.run.smithi019.stdout:(32/145): ceph-mgr-rook-18.2.1-341.g4e3182a6.el 673 kB/s | 50 kB 00:00 2024-01-24T14:16:07.695 INFO:teuthology.orchestra.run.smithi123.stdout:(121/145): openblas-0.3.15-6.el8.x86_64.rpm 15 MB/s | 5.0 MB 00:00 2024-01-24T14:16:07.715 INFO:teuthology.orchestra.run.smithi194.stdout:(90/145): librdkafka-0.11.4-3.el8.x86_64.rpm 2.4 MB/s | 354 kB 00:00 2024-01-24T14:16:07.720 INFO:teuthology.orchestra.run.smithi123.stdout:(122/145): libpq-13.11-1.el8.x86_64.rpm 1.4 MB/s | 198 kB 00:00 2024-01-24T14:16:07.741 INFO:teuthology.orchestra.run.smithi019.stdout:(33/145): ceph-prometheus-alerts-18.2.1-341.g4e 174 kB/s | 15 kB 00:00 2024-01-24T14:16:07.754 INFO:teuthology.orchestra.run.smithi123.stdout:(123/145): automake-1.16.1-8.el8.noarch.rpm 3.6 MB/s | 714 kB 00:00 2024-01-24T14:16:07.819 INFO:teuthology.orchestra.run.smithi123.stdout:(124/145): jq-1.6-7.el8.x86_64.rpm 1.6 MB/s | 202 kB 00:00 2024-01-24T14:16:07.833 INFO:teuthology.orchestra.run.smithi019.stdout:(34/145): ceph-volume-18.2.1-341.g4e3182a6.el8. 2.8 MB/s | 262 kB 00:00 2024-01-24T14:16:07.840 INFO:teuthology.orchestra.run.smithi194.stdout:(91/145): openblas-threads-0.3.15-6.el8.x86_64. 21 MB/s | 5.2 MB 00:00 2024-01-24T14:16:07.846 INFO:teuthology.orchestra.run.smithi123.stdout:(125/145): python3-pytz-2017.2-11.el8.noarch.rp 433 kB/s | 54 kB 00:00 2024-01-24T14:16:07.865 INFO:teuthology.orchestra.run.smithi194.stdout:(92/145): python3-mako-1.0.6-14.el8.noarch.rpm 1.0 MB/s | 157 kB 00:00 2024-01-24T14:16:07.917 INFO:teuthology.orchestra.run.smithi019.stdout:(35/145): cephadm-18.2.1-341.g4e3182a6.el8.noar 2.0 MB/s | 171 kB 00:00 2024-01-24T14:16:07.938 INFO:teuthology.orchestra.run.smithi123.stdout:(126/145): postgresql-10.23-3.module+el8.9.0+20 8.2 MB/s | 1.5 MB 00:00 2024-01-24T14:16:07.980 INFO:teuthology.orchestra.run.smithi123.stdout:(127/145): postgresql-contrib-10.23-3.module+el 4.9 MB/s | 811 kB 00:00 2024-01-24T14:16:08.001 INFO:teuthology.orchestra.run.smithi019.stdout:(36/145): cephfs-top-18.2.1-341.g4e3182a6.el8.n 216 kB/s | 18 kB 00:00 2024-01-24T14:16:08.041 INFO:teuthology.orchestra.run.smithi194.stdout:(93/145): openblas-0.3.15-6.el8.x86_64.rpm 25 MB/s | 5.0 MB 00:00 2024-01-24T14:16:08.099 INFO:teuthology.orchestra.run.smithi194.stdout:(94/145): automake-1.16.1-8.el8.noarch.rpm 3.0 MB/s | 714 kB 00:00 2024-01-24T14:16:08.168 INFO:teuthology.orchestra.run.smithi019.stdout:(37/145): ceph-mgr-diskprediction-local-18.2.1- 8.2 MB/s | 7.4 MB 00:00 2024-01-24T14:16:08.172 INFO:teuthology.orchestra.run.smithi123.stdout:(128/145): postgresql-server-10.23-3.module+el8 16 MB/s | 5.1 MB 00:00 2024-01-24T14:16:08.414 INFO:teuthology.orchestra.run.smithi123.stdout:(129/145): python3-scipy-1.0.0-21.module+el8.9. 29 MB/s | 14 MB 00:00 2024-01-24T14:16:08.443 INFO:teuthology.orchestra.run.smithi019.stdout:(38/145): liboath-2.6.2-3.el8.x86_64.rpm 218 kB/s | 59 kB 00:00 2024-01-24T14:16:08.502 INFO:teuthology.orchestra.run.smithi019.stdout:(39/145): gperftools-libs-2.7-9.el8.x86_64.rpm 612 kB/s | 306 kB 00:00 2024-01-24T14:16:08.519 INFO:teuthology.orchestra.run.smithi019.stdout:(40/145): libunwind-1.3.1-3.el8.x86_64.rpm 999 kB/s | 75 kB 00:00 2024-01-24T14:16:08.533 INFO:teuthology.orchestra.run.smithi194.stdout:(95/145): ceph-test-18.2.1-341.g4e3182a6.el8.x8 5.2 MB/s | 47 MB 00:09 2024-01-24T14:16:08.558 INFO:teuthology.orchestra.run.smithi194.stdout:(96/145): libpq-13.11-1.el8.x86_64.rpm 383 kB/s | 198 kB 00:00 2024-01-24T14:16:08.592 INFO:teuthology.orchestra.run.smithi194.stdout:(97/145): jq-1.6-7.el8.x86_64.rpm 411 kB/s | 202 kB 00:00 2024-01-24T14:16:08.594 INFO:teuthology.orchestra.run.smithi019.stdout:(41/145): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 582 kB/s | 44 kB 00:00 2024-01-24T14:16:08.661 INFO:teuthology.orchestra.run.smithi019.stdout:(42/145): python3-asyncssh-2.7.0-2.el8.noarch.r 2.9 MB/s | 476 kB 00:00 2024-01-24T14:16:08.681 INFO:teuthology.orchestra.run.smithi123.stdout:(130/145): librados2-18.2.1-341.g4e3182a6.el8.x 5.2 MB/s | 3.7 MB 00:00 2024-01-24T14:16:08.711 INFO:teuthology.orchestra.run.smithi019.stdout:(43/145): python3-beautifulsoup4-4.6.3-2.el8.1. 1.5 MB/s | 185 kB 00:00 2024-01-24T14:16:08.714 INFO:teuthology.orchestra.run.smithi123.stdout:(131/145): elfutils-libelf-0.189-3.el8.x86_64.r 773 kB/s | 232 kB 00:00 2024-01-24T14:16:08.733 INFO:teuthology.orchestra.run.smithi194.stdout:(98/145): postgresql-contrib-10.23-3.module+el8 5.6 MB/s | 811 kB 00:00 2024-01-24T14:16:08.737 INFO:teuthology.orchestra.run.smithi019.stdout:(44/145): python3-cachetools-3.1.1-4.el8.noarch 439 kB/s | 33 kB 00:00 2024-01-24T14:16:08.759 INFO:teuthology.orchestra.run.smithi194.stdout:(99/145): python3-pytz-2017.2-11.el8.noarch.rpm 241 kB/s | 54 kB 00:00 2024-01-24T14:16:08.787 INFO:teuthology.orchestra.run.smithi019.stdout:(45/145): python3-certifi-2018.10.15-7.el8.noar 217 kB/s | 16 kB 00:00 2024-01-24T14:16:08.812 INFO:teuthology.orchestra.run.smithi019.stdout:(46/145): python3-cheroot-8.5.2-1.el8.noarch.rp 2.2 MB/s | 173 kB 00:00 2024-01-24T14:16:08.834 INFO:teuthology.orchestra.run.smithi194.stdout:(100/145): postgresql-10.23-3.module+el8.9.0+20 5.6 MB/s | 1.5 MB 00:00 2024-01-24T14:16:08.887 INFO:teuthology.orchestra.run.smithi019.stdout:(47/145): python3-google-auth-1.1.1-10.el8.noar 1.1 MB/s | 81 kB 00:00 2024-01-24T14:16:08.921 INFO:teuthology.orchestra.run.smithi019.stdout:(48/145): python3-cherrypy-18.4.0-1.el8.noarch. 2.8 MB/s | 384 kB 00:00 2024-01-24T14:16:08.982 INFO:teuthology.orchestra.run.smithi123.stdout:(132/145): elfutils-libs-0.189-3.el8.x86_64.rpm 1.0 MB/s | 303 kB 00:00 2024-01-24T14:16:09.001 INFO:teuthology.orchestra.run.smithi194.stdout:(101/145): postgresql-server-10.23-3.module+el8 19 MB/s | 5.1 MB 00:00 2024-01-24T14:16:09.026 INFO:teuthology.orchestra.run.smithi194.stdout:(102/145): libverto-devel-0.3.0-5.el8.x86_64.rp 93 kB/s | 18 kB 00:00 2024-01-24T14:16:09.032 INFO:teuthology.orchestra.run.smithi123.stdout:(133/145): librbd1-18.2.1-341.g4e3182a6.el8.x86 4.1 MB/s | 3.6 MB 00:00 2024-01-24T14:16:09.057 INFO:teuthology.orchestra.run.smithi123.stdout:(134/145): libmount-2.32.1-43.el8.x86_64.rpm 691 kB/s | 237 kB 00:00 2024-01-24T14:16:09.160 INFO:teuthology.orchestra.run.smithi194.stdout:(103/145): python3-jwt-1.6.1-2.el8.noarch.rpm 269 kB/s | 43 kB 00:00 2024-01-24T14:16:09.185 INFO:teuthology.orchestra.run.smithi194.stdout:(104/145): python3-oauthlib-2.1.0-1.el8.noarch. 978 kB/s | 155 kB 00:00 2024-01-24T14:16:09.248 INFO:teuthology.orchestra.run.smithi123.stdout:(135/145): libsmartcols-2.32.1-43.el8.x86_64.rp 674 kB/s | 179 kB 00:00 2024-01-24T14:16:09.291 INFO:teuthology.orchestra.run.smithi123.stdout:(136/145): libuuid-2.32.1-43.el8.x86_64.rpm 383 kB/s | 99 kB 00:00 2024-01-24T14:16:09.399 INFO:teuthology.orchestra.run.smithi123.stdout:(137/145): util-linux-2.32.1-43.el8.x86_64.rpm 7.2 MB/s | 2.5 MB 00:00 2024-01-24T14:16:09.405 INFO:teuthology.orchestra.run.smithi019.stdout:(49/145): ceph-test-18.2.1-341.g4e3182a6.el8.x8 12 MB/s | 47 MB 00:03 2024-01-24T14:16:09.422 INFO:teuthology.orchestra.run.smithi019.stdout:(50/145): python3-jaraco-6.2-6.el8.noarch.rpm 21 kB/s | 11 kB 00:00 2024-01-24T14:16:09.427 INFO:teuthology.orchestra.run.smithi194.stdout:(105/145): python3-scipy-1.0.0-21.module+el8.9. 21 MB/s | 14 MB 00:00 2024-01-24T14:16:09.438 INFO:teuthology.orchestra.run.smithi019.stdout:(51/145): python3-jaraco-functools-2.0-4.el8.no 34 kB/s | 18 kB 00:00 2024-01-24T14:16:09.452 INFO:teuthology.orchestra.run.smithi194.stdout:(106/145): python3-requests-oauthlib-1.0.0-1.el 160 kB/s | 43 kB 00:00 2024-01-24T14:16:09.469 INFO:teuthology.orchestra.run.smithi194.stdout:(107/145): python3-pycparser-2.14-14.el8.noarch 353 kB/s | 109 kB 00:00 2024-01-24T14:16:09.489 INFO:teuthology.orchestra.run.smithi019.stdout:(52/145): python3-logutils-0.3.5-11.el8.noarch. 728 kB/s | 49 kB 00:00 2024-01-24T14:16:09.558 INFO:teuthology.orchestra.run.smithi123.stdout:(138/145): util-linux-user-2.32.1-43.el8.x86_64 331 kB/s | 103 kB 00:00 2024-01-24T14:16:09.572 INFO:teuthology.orchestra.run.smithi019.stdout:(53/145): python3-natsort-7.1.1-2.el8.noarch.rp 769 kB/s | 60 kB 00:00 2024-01-24T14:16:09.622 INFO:teuthology.orchestra.run.smithi194.stdout:(108/145): libaio-devel-0.3.112-1.el8.x86_64.rp 96 kB/s | 19 kB 00:00 2024-01-24T14:16:09.653 INFO:teuthology.orchestra.run.smithi194.stdout:(109/145): libzstd-devel-1.4.4-1.el8.x86_64.rpm 219 kB/s | 44 kB 00:00 2024-01-24T14:16:09.681 INFO:teuthology.orchestra.run.smithi019.stdout:(54/145): python3-more-itertools-7.2.0-3.el8.no 242 kB/s | 59 kB 00:00 2024-01-24T14:16:09.686 INFO:teuthology.orchestra.run.smithi194.stdout:(110/145): pcre2-devel-10.32-2.el8.x86_64.rpm 2.7 MB/s | 605 kB 00:00 2024-01-24T14:16:09.700 INFO:teuthology.orchestra.run.smithi123.stdout:(139/145): iproute-6.2.0-5.el8_9.x86_64.rpm 2.0 MB/s | 854 kB 00:00 2024-01-24T14:16:09.706 INFO:teuthology.orchestra.run.smithi019.stdout:(55/145): python3-pecan-1.3.2-9.el8.noarch.rpm 2.1 MB/s | 283 kB 00:00 2024-01-24T14:16:09.770 INFO:teuthology.orchestra.run.smithi194.stdout:(111/145): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.5 MB/s | 229 kB 00:00 2024-01-24T14:16:09.773 INFO:teuthology.orchestra.run.smithi019.stdout:(56/145): python3-portend-2.6-1.el8.noarch.rpm 170 kB/s | 16 kB 00:00 2024-01-24T14:16:09.790 INFO:teuthology.orchestra.run.smithi019.stdout:(57/145): python3-repoze-lru-0.7-6.el8.noarch.r 402 kB/s | 34 kB 00:00 2024-01-24T14:16:09.795 INFO:teuthology.orchestra.run.smithi194.stdout:(112/145): libselinux-devel-2.9-5.el8.x86_64.rp 1.8 MB/s | 200 kB 00:00 2024-01-24T14:16:09.824 INFO:teuthology.orchestra.run.smithi019.stdout:(58/145): python3-kubernetes-11.0.0-6.el8.noarc 2.8 MB/s | 1.2 MB 00:00 2024-01-24T14:16:09.825 INFO:teuthology.orchestra.run.smithi123.stdout:(140/145): libblkid-2.32.1-43.el8.x86_64.rpm 518 kB/s | 220 kB 00:00 2024-01-24T14:16:09.829 INFO:teuthology.orchestra.run.smithi194.stdout:(113/145): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.2 MB/s | 220 kB 00:00 2024-01-24T14:16:09.849 INFO:teuthology.orchestra.run.smithi019.stdout:(59/145): python3-rsa-4.9-2.el8.noarch.rpm 1.1 MB/s | 64 kB 00:00 2024-01-24T14:16:09.908 INFO:teuthology.orchestra.run.smithi019.stdout:(60/145): python3-simplegeneric-0.8.1-17.el8.no 224 kB/s | 19 kB 00:00 2024-01-24T14:16:09.909 INFO:teuthology.orchestra.run.smithi123.stdout:(141/145): libfdisk-2.32.1-43.el8.x86_64.rpm 724 kB/s | 254 kB 00:00 2024-01-24T14:16:09.933 INFO:teuthology.orchestra.run.smithi019.stdout:(61/145): python3-singledispatch-3.4.0.3-18.el8 286 kB/s | 24 kB 00:00 2024-01-24T14:16:09.959 INFO:teuthology.orchestra.run.smithi019.stdout:(62/145): python3-routes-2.4.1-12.el8.noarch.rp 1.0 MB/s | 196 kB 00:00 2024-01-24T14:16:09.992 INFO:teuthology.orchestra.run.smithi019.stdout:(63/145): python3-tempora-1.14.1-5.el8.noarch.r 349 kB/s | 29 kB 00:00 2024-01-24T14:16:09.993 INFO:teuthology.orchestra.run.smithi194.stdout:(114/145): keyutils-libs-devel-1.5.10-9.el8.x86 215 kB/s | 48 kB 00:00 2024-01-24T14:16:10.017 INFO:teuthology.orchestra.run.smithi019.stdout:(64/145): python3-trustme-0.6.0-4.el8.noarch.rp 329 kB/s | 27 kB 00:00 2024-01-24T14:16:10.021 INFO:teuthology.orchestra.run.smithi194.stdout:(115/145): zlib-devel-1.2.11-18.el8_5.x86_64.rp 300 kB/s | 58 kB 00:00 2024-01-24T14:16:10.038 INFO:teuthology.orchestra.run.smithi194.stdout:(116/145): libsepol-devel-2.9-3.el8.x86_64.rpm 359 kB/s | 87 kB 00:00 2024-01-24T14:16:10.042 INFO:teuthology.orchestra.run.smithi019.stdout:(65/145): python3-waitress-1.4.3-1.el8.noarch.r 3.0 MB/s | 255 kB 00:00 2024-01-24T14:16:10.058 INFO:teuthology.orchestra.run.smithi123.stdout:(142/145): xfsprogs-5.0.0-12.el8.x86_64.rpm 3.0 MB/s | 1.1 MB 00:00 2024-01-24T14:16:10.088 INFO:teuthology.orchestra.run.smithi194.stdout:(117/145): fuse-2.9.7-17.el8.x86_64.rpm 1.2 MB/s | 83 kB 00:00 2024-01-24T14:16:10.093 INFO:teuthology.orchestra.run.smithi019.stdout:(66/145): python3-websocket-client-0.56.0-5.el8 816 kB/s | 61 kB 00:00 2024-01-24T14:16:10.118 INFO:teuthology.orchestra.run.smithi019.stdout:(67/145): python3-webob-1.8.5-1.el8.1.noarch.rp 2.0 MB/s | 251 kB 00:00 2024-01-24T14:16:10.139 INFO:teuthology.orchestra.run.smithi194.stdout:(118/145): libcom_err-devel-1.45.6-4.el8.x86_64 267 kB/s | 39 kB 00:00 2024-01-24T14:16:10.169 INFO:teuthology.orchestra.run.smithi019.stdout:(68/145): python3-webtest-2.0.33-1.el8.noarch.r 680 kB/s | 85 kB 00:00 2024-01-24T14:16:10.176 INFO:teuthology.orchestra.run.smithi123.stdout:(143/145): openssl-libs-1.1.1k-12.el8_9.x86_64. 5.5 MB/s | 1.5 MB 00:00 2024-01-24T14:16:10.181 INFO:teuthology.orchestra.run.smithi194.stdout:(119/145): libgfortran-8.5.0-20.el8.x86_64.rpm 4.4 MB/s | 645 kB 00:00 2024-01-24T14:16:10.194 INFO:teuthology.orchestra.run.smithi019.stdout:(69/145): python3-zc-lockfile-2.0-2.el8.noarch. 228 kB/s | 23 kB 00:00 2024-01-24T14:16:10.209 INFO:teuthology.orchestra.run.smithi123.stdout:(144/145): openssl-1.1.1k-12.el8_9.x86_64.rpm 1.8 MB/s | 711 kB 00:00 2024-01-24T14:16:10.211 INFO:teuthology.orchestra.run.smithi019.stdout:(70/145): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.6 MB/s | 70 kB 00:00 2024-01-24T14:16:10.231 INFO:teuthology.orchestra.run.smithi194.stdout:(120/145): libquadmath-8.5.0-20.el8.x86_64.rpm 1.2 MB/s | 172 kB 00:00 2024-01-24T14:16:10.243 INFO:teuthology.orchestra.run.smithi123.stdout:(145/145): xfsdump-3.1.8-7.el8_9.x86_64.rpm 1.8 MB/s | 333 kB 00:00 2024-01-24T14:16:10.246 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-01-24T14:16:10.246 INFO:teuthology.orchestra.run.smithi123.stdout:Total 13 MB/s | 212 MB 00:16 2024-01-24T14:16:10.323 INFO:teuthology.orchestra.run.smithi194.stdout:(121/145): libuuid-devel-2.32.1-43.el8.x86_64.r 543 kB/s | 100 kB 00:00 2024-01-24T14:16:10.378 INFO:teuthology.orchestra.run.smithi019.stdout:(71/145): python3-prettytable-0.7.2-14.el8.noar 264 kB/s | 44 kB 00:00 2024-01-24T14:16:10.390 INFO:teuthology.orchestra.run.smithi194.stdout:(122/145): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.1 MB/s | 238 kB 00:00 2024-01-24T14:16:10.403 INFO:teuthology.orchestra.run.smithi019.stdout:(72/145): python3-jsonpointer-1.10-11.el8.noarc 96 kB/s | 20 kB 00:00 2024-01-24T14:16:10.424 INFO:teuthology.orchestra.run.smithi194.stdout:(123/145): python3-cryptography-3.2.1-6.el8.x86 2.8 MB/s | 559 kB 00:00 2024-01-24T14:16:10.479 INFO:teuthology.orchestra.run.smithi019.stdout:(73/145): python3-pyasn1-modules-0.3.7-6.el8.no 1.1 MB/s | 110 kB 00:00 2024-01-24T14:16:10.482 INFO:teuthology.orchestra.run.smithi194.stdout:(124/145): xfsprogs-devel-5.0.0-12.el8.x86_64.r 456 kB/s | 73 kB 00:00 2024-01-24T14:16:10.544 INFO:teuthology.orchestra.run.smithi019.stdout:(74/145): thrift-0.13.0-2.el8.x86_64.rpm 4.1 MB/s | 1.7 MB 00:00 2024-01-24T14:16:10.549 INFO:teuthology.orchestra.run.smithi194.stdout:(125/145): librabbitmq-0.9.0-4.el8.x86_64.rpm 376 kB/s | 47 kB 00:00 2024-01-24T14:16:10.583 INFO:teuthology.orchestra.run.smithi194.stdout:(126/145): elfutils-libelf-devel-0.189-3.el8.x8 321 kB/s | 62 kB 00:00 2024-01-24T14:16:10.641 INFO:teuthology.orchestra.run.smithi194.stdout:(127/145): krb5-devel-1.18.2-26.el8_9.x86_64.rp 3.5 MB/s | 562 kB 00:00 2024-01-24T14:16:10.750 INFO:teuthology.orchestra.run.smithi194.stdout:(128/145): openssl-devel-1.1.1k-12.el8_9.x86_64 12 MB/s | 2.3 MB 00:00 2024-01-24T14:16:10.775 INFO:teuthology.orchestra.run.smithi194.stdout:(129/145): NetworkManager-initscripts-updown-1. 750 kB/s | 144 kB 00:00 2024-01-24T14:16:10.836 INFO:teuthology.orchestra.run.smithi019.stdout:(75/145): python3-jsonpatch-1.21-2.el8.noarch.r 61 kB/s | 27 kB 00:00 2024-01-24T14:16:10.871 INFO:teuthology.orchestra.run.smithi019.stdout:(76/145): python3-pyasn1-0.3.7-6.el8.noarch.rpm 322 kB/s | 126 kB 00:00 2024-01-24T14:16:10.900 INFO:teuthology.orchestra.run.smithi194.stdout:(130/145): elfutils-libelf-0.189-3.el8.x86_64.r 1.8 MB/s | 232 kB 00:00 2024-01-24T14:16:10.963 INFO:teuthology.orchestra.run.smithi019.stdout:(77/145): python3-werkzeug-0.12.2-4.el8.noarch. 1.1 MB/s | 457 kB 00:00 2024-01-24T14:16:10.996 INFO:teuthology.orchestra.run.smithi019.stdout:(78/145): flex-2.6.1-9.el8.x86_64.rpm 2.5 MB/s | 320 kB 00:00 2024-01-24T14:16:11.038 INFO:teuthology.orchestra.run.smithi019.stdout:(79/145): bison-3.0.4-10.el8.x86_64.rpm 3.5 MB/s | 688 kB 00:00 2024-01-24T14:16:11.042 INFO:teuthology.orchestra.run.smithi194.stdout:(131/145): elfutils-libs-0.189-3.el8.x86_64.rpm 2.1 MB/s | 303 kB 00:00 2024-01-24T14:16:11.097 INFO:teuthology.orchestra.run.smithi019.stdout:(80/145): python3-markupsafe-0.23-19.el8.x86_64 387 kB/s | 39 kB 00:00 2024-01-24T14:16:11.111 INFO:teuthology.orchestra.run.smithi194.stdout:(132/145): librados2-18.2.1-341.g4e3182a6.el8.x 7.8 MB/s | 3.7 MB 00:00 2024-01-24T14:16:11.131 INFO:teuthology.orchestra.run.smithi019.stdout:(81/145): libtool-2.4.6-25.el8.x86_64.rpm 4.1 MB/s | 709 kB 00:00 2024-01-24T14:16:11.159 INFO:teuthology.orchestra.run.smithi194.stdout:(133/145): libmount-2.32.1-43.el8.x86_64.rpm 2.0 MB/s | 237 kB 00:00 2024-01-24T14:16:11.164 INFO:teuthology.orchestra.run.smithi019.stdout:(82/145): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.0 MB/s | 259 kB 00:00 2024-01-24T14:16:11.190 INFO:teuthology.orchestra.run.smithi019.stdout:(83/145): oniguruma-6.8.2-2.el8.x86_64.rpm 2.0 MB/s | 187 kB 00:00 2024-01-24T14:16:11.257 INFO:teuthology.orchestra.run.smithi019.stdout:(84/145): uuid-1.6.2-43.el8.x86_64.rpm 508 kB/s | 64 kB 00:00 2024-01-24T14:16:11.310 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-01-24T14:16:11.318 INFO:teuthology.orchestra.run.smithi194.stdout:(134/145): librbd1-18.2.1-341.g4e3182a6.el8.x86 6.3 MB/s | 3.6 MB 00:00 2024-01-24T14:16:11.343 INFO:teuthology.orchestra.run.smithi194.stdout:(135/145): libuuid-2.32.1-43.el8.x86_64.rpm 539 kB/s | 99 kB 00:00 2024-01-24T14:16:11.368 INFO:teuthology.orchestra.run.smithi194.stdout:(136/145): libsmartcols-2.32.1-43.el8.x86_64.rp 698 kB/s | 179 kB 00:00 2024-01-24T14:16:11.399 INFO:teuthology.orchestra.run.smithi019.stdout:(85/145): python3-babel-2.5.1-7.el8.noarch.rpm 23 MB/s | 4.8 MB 00:00 2024-01-24T14:16:11.424 INFO:teuthology.orchestra.run.smithi019.stdout:(86/145): python3-pyOpenSSL-19.0.0-1.el8.noarch 399 kB/s | 103 kB 00:00 2024-01-24T14:16:11.452 INFO:teuthology.orchestra.run.smithi194.stdout:(137/145): util-linux-2.32.1-43.el8.x86_64.rpm 19 MB/s | 2.5 MB 00:00 2024-01-24T14:16:11.457 INFO:teuthology.orchestra.run.smithi019.stdout:(87/145): python3-jinja2-2.10.1-3.el8.noarch.rp 2.6 MB/s | 538 kB 00:00 2024-01-24T14:16:11.511 INFO:teuthology.orchestra.run.smithi194.stdout:(138/145): iproute-6.2.0-5.el8_9.x86_64.rpm 5.9 MB/s | 854 kB 00:00 2024-01-24T14:16:11.536 INFO:teuthology.orchestra.run.smithi194.stdout:(139/145): util-linux-user-2.32.1-43.el8.x86_64 534 kB/s | 103 kB 00:00 2024-01-24T14:16:11.553 INFO:teuthology.orchestra.run.smithi194.stdout:(140/145): libblkid-2.32.1-43.el8.x86_64.rpm 2.1 MB/s | 220 kB 00:00 2024-01-24T14:16:11.600 INFO:teuthology.orchestra.run.smithi019.stdout:(88/145): python3-numpy-1.14.3-10.el8.x86_64.rp 19 MB/s | 3.7 MB 00:00 2024-01-24T14:16:11.625 INFO:teuthology.orchestra.run.smithi019.stdout:(89/145): python3-numpy-f2py-1.14.3-10.el8.x86_ 1.3 MB/s | 225 kB 00:00 2024-01-24T14:16:11.636 INFO:teuthology.orchestra.run.smithi194.stdout:(141/145): libfdisk-2.32.1-43.el8.x86_64.rpm 2.0 MB/s | 254 kB 00:00 2024-01-24T14:16:11.650 INFO:teuthology.orchestra.run.smithi019.stdout:(90/145): socat-1.7.4.1-1.el8.x86_64.rpm 1.4 MB/s | 323 kB 00:00 2024-01-24T14:16:11.705 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-01-24T14:16:11.705 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-01-24T14:16:11.706 INFO:teuthology.orchestra.run.smithi194.stdout:(142/145): openssl-1.1.1k-12.el8_9.x86_64.rpm 4.6 MB/s | 711 kB 00:00 2024-01-24T14:16:11.725 INFO:teuthology.orchestra.run.smithi019.stdout:(91/145): librdkafka-0.11.4-3.el8.x86_64.rpm 2.8 MB/s | 354 kB 00:00 2024-01-24T14:16:11.745 INFO:teuthology.orchestra.run.smithi194.stdout:(143/145): xfsprogs-5.0.0-12.el8.x86_64.rpm 5.1 MB/s | 1.1 MB 00:00 2024-01-24T14:16:11.784 INFO:teuthology.orchestra.run.smithi019.stdout:(92/145): python3-mako-1.0.6-14.el8.noarch.rpm 1.1 MB/s | 157 kB 00:00 2024-01-24T14:16:11.804 INFO:teuthology.orchestra.run.smithi194.stdout:(144/145): openssl-libs-1.1.1k-12.el8_9.x86_64. 8.8 MB/s | 1.5 MB 00:00 2024-01-24T14:16:11.879 INFO:teuthology.orchestra.run.smithi194.stdout:(145/145): xfsdump-3.1.8-7.el8_9.x86_64.rpm 1.9 MB/s | 333 kB 00:00 2024-01-24T14:16:11.882 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-01-24T14:16:11.882 INFO:teuthology.orchestra.run.smithi194.stdout:Total 12 MB/s | 212 MB 00:17 2024-01-24T14:16:11.909 INFO:teuthology.orchestra.run.smithi019.stdout:(93/145): openblas-threads-0.3.15-6.el8.x86_64. 18 MB/s | 5.2 MB 00:00 2024-01-24T14:16:11.951 INFO:teuthology.orchestra.run.smithi019.stdout:(94/145): automake-1.16.1-8.el8.noarch.rpm 4.2 MB/s | 714 kB 00:00 2024-01-24T14:16:12.043 INFO:teuthology.orchestra.run.smithi019.stdout:(95/145): openblas-0.3.15-6.el8.x86_64.rpm 16 MB/s | 5.0 MB 00:00 2024-01-24T14:16:12.097 INFO:teuthology.orchestra.run.smithi019.stdout:(96/145): libpq-13.11-1.el8.x86_64.rpm 1.0 MB/s | 198 kB 00:00 2024-01-24T14:16:12.127 INFO:teuthology.orchestra.run.smithi019.stdout:(97/145): jq-1.6-7.el8.x86_64.rpm 1.1 MB/s | 202 kB 00:00 2024-01-24T14:16:12.161 INFO:teuthology.orchestra.run.smithi019.stdout:(98/145): python3-pytz-2017.2-11.el8.noarch.rpm 463 kB/s | 54 kB 00:00 2024-01-24T14:16:12.261 INFO:teuthology.orchestra.run.smithi019.stdout:(99/145): postgresql-10.23-3.module+el8.9.0+208 9.2 MB/s | 1.5 MB 00:00 2024-01-24T14:16:12.303 INFO:teuthology.orchestra.run.smithi019.stdout:(100/145): postgresql-contrib-10.23-3.module+el 4.5 MB/s | 811 kB 00:00 2024-01-24T14:16:12.428 INFO:teuthology.orchestra.run.smithi019.stdout:(101/145): postgresql-server-10.23-3.module+el8 19 MB/s | 5.1 MB 00:00 2024-01-24T14:16:12.483 INFO:teuthology.orchestra.run.smithi019.stdout:(102/145): libverto-devel-0.3.0-5.el8.x86_64.rp 99 kB/s | 18 kB 00:00 2024-01-24T14:16:12.604 INFO:teuthology.orchestra.run.smithi019.stdout:(103/145): python3-jwt-1.6.1-2.el8.noarch.rpm 243 kB/s | 43 kB 00:00 2024-01-24T14:16:12.629 INFO:teuthology.orchestra.run.smithi019.stdout:(104/145): python3-oauthlib-2.1.0-1.el8.noarch. 1.0 MB/s | 155 kB 00:00 2024-01-24T14:16:12.704 INFO:teuthology.orchestra.run.smithi019.stdout:(105/145): python3-pycparser-2.14-14.el8.noarch 1.1 MB/s | 109 kB 00:00 2024-01-24T14:16:12.725 INFO:teuthology.orchestra.run.smithi019.stdout:(106/145): python3-requests-oauthlib-1.0.0-1.el 464 kB/s | 43 kB 00:00 2024-01-24T14:16:12.884 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-01-24T14:16:12.888 INFO:teuthology.orchestra.run.smithi019.stdout:(107/145): python3-scipy-1.0.0-21.module+el8.9. 22 MB/s | 14 MB 00:00 2024-01-24T14:16:12.894 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-01-24T14:16:12.913 INFO:teuthology.orchestra.run.smithi019.stdout:(108/145): libaio-devel-0.3.112-1.el8.x86_64.rp 89 kB/s | 19 kB 00:00 2024-01-24T14:16:12.938 INFO:teuthology.orchestra.run.smithi019.stdout:(109/145): libzstd-devel-1.4.4-1.el8.x86_64.rpm 202 kB/s | 44 kB 00:00 2024-01-24T14:16:12.948 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-01-24T14:16:13.031 INFO:teuthology.orchestra.run.smithi019.stdout:(110/145): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.9 MB/s | 229 kB 00:00 2024-01-24T14:16:13.064 INFO:teuthology.orchestra.run.smithi019.stdout:(111/145): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.7 MB/s | 220 kB 00:00 2024-01-24T14:16:13.098 INFO:teuthology.orchestra.run.smithi019.stdout:(112/145): pcre2-devel-10.32-2.el8.x86_64.rpm 2.8 MB/s | 605 kB 00:00 2024-01-24T14:16:13.190 INFO:teuthology.orchestra.run.smithi019.stdout:(113/145): libselinux-devel-2.9-5.el8.x86_64.rp 1.2 MB/s | 200 kB 00:00 2024-01-24T14:16:13.216 INFO:teuthology.orchestra.run.smithi019.stdout:(114/145): keyutils-libs-devel-1.5.10-9.el8.x86 319 kB/s | 48 kB 00:00 2024-01-24T14:16:13.282 INFO:teuthology.orchestra.run.smithi019.stdout:(115/145): libsepol-devel-2.9-3.el8.x86_64.rpm 473 kB/s | 87 kB 00:00 2024-01-24T14:16:13.324 INFO:teuthology.orchestra.run.smithi019.stdout:(116/145): zlib-devel-1.2.11-18.el8_5.x86_64.rp 431 kB/s | 58 kB 00:00 2024-01-24T14:16:13.344 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-01-24T14:16:13.344 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-01-24T14:16:13.408 INFO:teuthology.orchestra.run.smithi019.stdout:(117/145): libcom_err-devel-1.45.6-4.el8.x86_64 201 kB/s | 39 kB 00:00 2024-01-24T14:16:13.466 INFO:teuthology.orchestra.run.smithi019.stdout:(118/145): fuse-2.9.7-17.el8.x86_64.rpm 452 kB/s | 83 kB 00:00 2024-01-24T14:16:13.544 INFO:teuthology.orchestra.run.smithi019.stdout:(119/145): libquadmath-8.5.0-20.el8.x86_64.rpm 1.3 MB/s | 172 kB 00:00 2024-01-24T14:16:13.584 INFO:teuthology.orchestra.run.smithi019.stdout:(120/145): libgfortran-8.5.0-20.el8.x86_64.rpm 2.4 MB/s | 645 kB 00:00 2024-01-24T14:16:13.609 INFO:teuthology.orchestra.run.smithi019.stdout:(121/145): libuuid-devel-2.32.1-43.el8.x86_64.r 701 kB/s | 100 kB 00:00 2024-01-24T14:16:13.676 INFO:teuthology.orchestra.run.smithi019.stdout:(122/145): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.7 MB/s | 238 kB 00:00 2024-01-24T14:16:13.710 INFO:teuthology.orchestra.run.smithi019.stdout:(123/145): xfsprogs-devel-5.0.0-12.el8.x86_64.r 723 kB/s | 73 kB 00:00 2024-01-24T14:16:13.735 INFO:teuthology.orchestra.run.smithi019.stdout:(124/145): python3-cryptography-3.2.1-6.el8.x86 3.6 MB/s | 559 kB 00:00 2024-01-24T14:16:13.794 INFO:teuthology.orchestra.run.smithi019.stdout:(125/145): elfutils-libelf-devel-0.189-3.el8.x8 526 kB/s | 62 kB 00:00 2024-01-24T14:16:13.852 INFO:teuthology.orchestra.run.smithi019.stdout:(126/145): krb5-devel-1.18.2-26.el8_9.x86_64.rp 4.7 MB/s | 562 kB 00:00 2024-01-24T14:16:13.886 INFO:teuthology.orchestra.run.smithi019.stdout:(127/145): librabbitmq-0.9.0-4.el8.x86_64.rpm 267 kB/s | 47 kB 00:00 2024-01-24T14:16:13.928 INFO:teuthology.orchestra.run.smithi019.stdout:(128/145): NetworkManager-initscripts-updown-1. 1.9 MB/s | 144 kB 00:00 2024-01-24T14:16:13.969 INFO:teuthology.orchestra.run.smithi019.stdout:(129/145): openssl-devel-1.1.1k-12.el8_9.x86_64 13 MB/s | 2.3 MB 00:00 2024-01-24T14:16:14.112 INFO:teuthology.orchestra.run.smithi019.stdout:(130/145): elfutils-libelf-0.189-3.el8.x86_64.r 1.6 MB/s | 232 kB 00:00 2024-01-24T14:16:14.262 INFO:teuthology.orchestra.run.smithi019.stdout:(131/145): elfutils-libs-0.189-3.el8.x86_64.rpm 2.0 MB/s | 303 kB 00:00 2024-01-24T14:16:14.321 INFO:teuthology.orchestra.run.smithi019.stdout:(132/145): librbd1-18.2.1-341.g4e3182a6.el8.x86 9.1 MB/s | 3.6 MB 00:00 2024-01-24T14:16:14.379 INFO:teuthology.orchestra.run.smithi019.stdout:(133/145): librados2-18.2.1-341.g4e3182a6.el8.x 7.4 MB/s | 3.7 MB 00:00 2024-01-24T14:16:14.438 INFO:teuthology.orchestra.run.smithi019.stdout:(134/145): libmount-2.32.1-43.el8.x86_64.rpm 1.3 MB/s | 237 kB 00:00 2024-01-24T14:16:14.463 INFO:teuthology.orchestra.run.smithi019.stdout:(135/145): libsmartcols-2.32.1-43.el8.x86_64.rp 1.2 MB/s | 179 kB 00:00 2024-01-24T14:16:14.505 INFO:teuthology.orchestra.run.smithi019.stdout:(136/145): libuuid-2.32.1-43.el8.x86_64.rpm 788 kB/s | 99 kB 00:00 2024-01-24T14:16:14.532 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-01-24T14:16:14.542 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-01-24T14:16:14.597 INFO:teuthology.orchestra.run.smithi019.stdout:(137/145): util-linux-user-2.32.1-43.el8.x86_64 769 kB/s | 103 kB 00:00 2024-01-24T14:16:14.689 INFO:teuthology.orchestra.run.smithi019.stdout:(138/145): util-linux-2.32.1-43.el8.x86_64.rpm 9.9 MB/s | 2.5 MB 00:00 2024-01-24T14:16:14.731 INFO:teuthology.orchestra.run.smithi019.stdout:(139/145): iproute-6.2.0-5.el8_9.x86_64.rpm 3.7 MB/s | 854 kB 00:00 2024-01-24T14:16:14.765 INFO:teuthology.orchestra.run.smithi019.stdout:(140/145): libblkid-2.32.1-43.el8.x86_64.rpm 1.3 MB/s | 220 kB 00:00 2024-01-24T14:16:14.823 INFO:teuthology.orchestra.run.smithi019.stdout:(141/145): libfdisk-2.32.1-43.el8.x86_64.rpm 1.9 MB/s | 254 kB 00:00 2024-01-24T14:16:14.907 INFO:teuthology.orchestra.run.smithi019.stdout:(142/145): openssl-1.1.1k-12.el8_9.x86_64.rpm 4.9 MB/s | 711 kB 00:00 2024-01-24T14:16:14.940 INFO:teuthology.orchestra.run.smithi019.stdout:(143/145): xfsprogs-5.0.0-12.el8.x86_64.rpm 5.1 MB/s | 1.1 MB 00:00 2024-01-24T14:16:15.007 INFO:teuthology.orchestra.run.smithi019.stdout:(144/145): openssl-libs-1.1.1k-12.el8_9.x86_64. 8.0 MB/s | 1.5 MB 00:00 2024-01-24T14:16:15.082 INFO:teuthology.orchestra.run.smithi019.stdout:(145/145): xfsdump-3.1.8-7.el8_9.x86_64.rpm 1.9 MB/s | 333 kB 00:00 2024-01-24T14:16:15.088 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-01-24T14:16:15.088 INFO:teuthology.orchestra.run.smithi019.stdout:Total 15 MB/s | 212 MB 00:14 2024-01-24T14:16:16.207 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-01-24T14:16:16.660 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-01-24T14:16:16.660 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-01-24T14:16:17.968 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-01-24T14:16:17.980 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-01-24T14:16:19.015 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-01-24T14:16:19.150 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/1 2024-01-24T14:16:19.447 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/161 2024-01-24T14:16:19.665 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/161 2024-01-24T14:16:19.889 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/161 2024-01-24T14:16:20.091 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : libuuid-2.32.1-43.el8.x86_64 3/161 2024-01-24T14:16:20.169 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libuuid-2.32.1-43.el8.x86_64 3/161 2024-01-24T14:16:20.176 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-01-24T14:16:20.311 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/1 2024-01-24T14:16:20.356 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : libblkid-2.32.1-43.el8.x86_64 4/161 2024-01-24T14:16:20.447 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libblkid-2.32.1-43.el8.x86_64 4/161 2024-01-24T14:16:20.605 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/161 2024-01-24T14:16:20.669 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/161 2024-01-24T14:16:20.826 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/161 2024-01-24T14:16:20.888 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 6/161 2024-01-24T14:16:21.021 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/161 2024-01-24T14:16:21.106 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 7/161 2024-01-24T14:16:21.204 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 7/161 2024-01-24T14:16:21.232 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : libuuid-2.32.1-43.el8.x86_64 3/161 2024-01-24T14:16:21.321 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libuuid-2.32.1-43.el8.x86_64 3/161 2024-01-24T14:16:21.431 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 8/161 2024-01-24T14:16:21.743 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : libblkid-2.32.1-43.el8.x86_64 4/161 2024-01-24T14:16:21.827 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 8/161 2024-01-24T14:16:21.937 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libblkid-2.32.1-43.el8.x86_64 4/161 2024-01-24T14:16:22.147 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/161 2024-01-24T14:16:22.374 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 6/161 2024-01-24T14:16:22.389 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : librados2-2:18.2.1-341.g4e3182a6.el8.x86_64 9/161 2024-01-24T14:16:22.434 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 6/161 2024-01-24T14:16:22.628 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/161 2024-01-24T14:16:22.771 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librados2-2:18.2.1-341.g4e3182a6.el8.x86_64 9/161 2024-01-24T14:16:22.832 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 8/161 2024-01-24T14:16:22.998 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : librbd1-2:18.2.1-341.g4e3182a6.el8.x86_64 10/161 2024-01-24T14:16:23.130 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librbd1-2:18.2.1-341.g4e3182a6.el8.x86_64 10/161 2024-01-24T14:16:23.221 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 8/161 2024-01-24T14:16:23.331 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libcephfs2-2:18.2.1-341.g4e3182a6.el8.x86_64 11/161 2024-01-24T14:16:23.456 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : librados2-2:18.2.1-341.g4e3182a6.el8.x86_64 9/161 2024-01-24T14:16:23.459 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libcephfs2-2:18.2.1-341.g4e3182a6.el8.x86_64 11/161 2024-01-24T14:16:23.791 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rados-2:18.2.1-341.g4e3182a6.el8.x86_64 12/161 2024-01-24T14:16:23.831 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-01-24T14:16:23.847 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librados2-2:18.2.1-341.g4e3182a6.el8.x86_64 9/161 2024-01-24T14:16:23.980 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/1 2024-01-24T14:16:24.057 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : librbd1-2:18.2.1-341.g4e3182a6.el8.x86_64 10/161 2024-01-24T14:16:24.097 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 13/161 2024-01-24T14:16:24.166 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 13/161 2024-01-24T14:16:24.180 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librbd1-2:18.2.1-341.g4e3182a6.el8.x86_64 10/161 2024-01-24T14:16:24.357 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/161 2024-01-24T14:16:24.381 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcephfs2-2:18.2.1-341.g4e3182a6.el8.x86_64 11/161 2024-01-24T14:16:24.387 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libpq-13.11-1.el8.x86_64 14/161 2024-01-24T14:16:24.476 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libcephfs2-2:18.2.1-341.g4e3182a6.el8.x86_64 11/161 2024-01-24T14:16:24.590 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/161 2024-01-24T14:16:24.660 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : elfutils-libelf-0.189-3.el8.x86_64 15/161 2024-01-24T14:16:24.767 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/161 2024-01-24T14:16:24.808 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rados-2:18.2.1-341.g4e3182a6.el8.x86_64 12/161 2024-01-24T14:16:24.876 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 16/161 2024-01-24T14:16:24.946 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : libuuid-2.32.1-43.el8.x86_64 3/161 2024-01-24T14:16:25.028 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libuuid-2.32.1-43.el8.x86_64 3/161 2024-01-24T14:16:25.098 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 13/161 2024-01-24T14:16:25.168 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 13/161 2024-01-24T14:16:25.252 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 17/161 2024-01-24T14:16:25.330 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : libblkid-2.32.1-43.el8.x86_64 4/161 2024-01-24T14:16:25.405 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libblkid-2.32.1-43.el8.x86_64 4/161 2024-01-24T14:16:25.437 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libpq-13.11-1.el8.x86_64 14/161 2024-01-24T14:16:25.518 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : postgresql-10.23-3.module+el8.9.0+20831+78489779 18/161 2024-01-24T14:16:25.615 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/161 2024-01-24T14:16:25.714 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libcephsqlite-2:18.2.1-341.g4e3182a6.el8.x86_64 19/161 2024-01-24T14:16:25.718 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : elfutils-libelf-0.189-3.el8.x86_64 15/161 2024-01-24T14:16:25.818 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libcephsqlite-2:18.2.1-341.g4e3182a6.el8.x86_64 19/161 2024-01-24T14:16:25.839 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 6/161 2024-01-24T14:16:25.914 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 6/161 2024-01-24T14:16:25.918 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 16/161 2024-01-24T14:16:25.997 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libradosstriper1-2:18.2.1-341.g4e3182a6.el8.x86_ 20/161 2024-01-24T14:16:26.072 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libradosstriper1-2:18.2.1-341.g4e3182a6.el8.x86_ 20/161 2024-01-24T14:16:26.096 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/161 2024-01-24T14:16:26.267 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 21/161 2024-01-24T14:16:26.271 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 8/161 2024-01-24T14:16:26.326 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 17/161 2024-01-24T14:16:26.498 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 22/161 2024-01-24T14:16:26.577 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : postgresql-10.23-3.module+el8.9.0+20831+78489779 18/161 2024-01-24T14:16:26.705 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 8/161 2024-01-24T14:16:26.710 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 23/161 2024-01-24T14:16:26.775 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcephsqlite-2:18.2.1-341.g4e3182a6.el8.x86_64 19/161 2024-01-24T14:16:26.868 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libcephsqlite-2:18.2.1-341.g4e3182a6.el8.x86_64 19/161 2024-01-24T14:16:26.897 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 24/161 2024-01-24T14:16:26.938 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : librados2-2:18.2.1-341.g4e3182a6.el8.x86_64 9/161 2024-01-24T14:16:27.012 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 24/161 2024-01-24T14:16:27.014 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libradosstriper1-2:18.2.1-341.g4e3182a6.el8.x86_ 20/161 2024-01-24T14:16:27.091 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libradosstriper1-2:18.2.1-341.g4e3182a6.el8.x86_ 20/161 2024-01-24T14:16:27.197 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 25/161 2024-01-24T14:16:27.314 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 21/161 2024-01-24T14:16:27.357 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librados2-2:18.2.1-341.g4e3182a6.el8.x86_64 9/161 2024-01-24T14:16:27.433 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 21/161 2024-01-24T14:16:27.522 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : librbd1-2:18.2.1-341.g4e3182a6.el8.x86_64 10/161 2024-01-24T14:16:27.606 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 22/161 2024-01-24T14:16:27.661 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librbd1-2:18.2.1-341.g4e3182a6.el8.x86_64 10/161 2024-01-24T14:16:27.847 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libcephfs2-2:18.2.1-341.g4e3182a6.el8.x86_64 11/161 2024-01-24T14:16:27.858 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 25/161 2024-01-24T14:16:27.954 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libcephfs2-2:18.2.1-341.g4e3182a6.el8.x86_64 11/161 2024-01-24T14:16:28.047 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 26/161 2024-01-24T14:16:28.215 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 22/161 2024-01-24T14:16:28.282 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rados-2:18.2.1-341.g4e3182a6.el8.x86_64 12/161 2024-01-24T14:16:28.381 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 23/161 2024-01-24T14:16:28.538 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 13/161 2024-01-24T14:16:28.628 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 13/161 2024-01-24T14:16:28.673 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 26/161 2024-01-24T14:16:28.815 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libpq-13.11-1.el8.x86_64 14/161 2024-01-24T14:16:28.855 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : openblas-0.3.15-6.el8.x86_64 27/161 2024-01-24T14:16:28.971 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 23/161 2024-01-24T14:16:29.071 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : elfutils-libelf-0.189-3.el8.x86_64 15/161 2024-01-24T14:16:29.164 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : openblas-0.3.15-6.el8.x86_64 24/161 2024-01-24T14:16:29.269 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 16/161 2024-01-24T14:16:29.561 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 27/161 2024-01-24T14:16:29.659 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 17/161 2024-01-24T14:16:29.827 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 24/161 2024-01-24T14:16:29.836 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 28/161 2024-01-24T14:16:29.886 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : postgresql-10.23-3.module+el8.9.0+20831+78489779 18/161 2024-01-24T14:16:30.019 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 29/161 2024-01-24T14:16:30.072 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libcephsqlite-2:18.2.1-341.g4e3182a6.el8.x86_64 19/161 2024-01-24T14:16:30.128 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 25/161 2024-01-24T14:16:30.181 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libcephsqlite-2:18.2.1-341.g4e3182a6.el8.x86_64 19/161 2024-01-24T14:16:30.286 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 26/161 2024-01-24T14:16:30.289 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 30/161 2024-01-24T14:16:30.321 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libradosstriper1-2:18.2.1-341.g4e3182a6.el8.x86_ 20/161 2024-01-24T14:16:30.433 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libradosstriper1-2:18.2.1-341.g4e3182a6.el8.x86_ 20/161 2024-01-24T14:16:30.501 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 27/161 2024-01-24T14:16:30.508 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-ceph-common-2:18.2.1-341.g4e3182a6.el8.x 31/161 2024-01-24T14:16:30.579 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 21/161 2024-01-24T14:16:30.690 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-ceph-argparse-2:18.2.1-341.g4e3182a6.el8 32/161 2024-01-24T14:16:30.728 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 21/161 2024-01-24T14:16:30.759 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 28/161 2024-01-24T14:16:30.887 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 22/161 2024-01-24T14:16:31.001 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cephfs-2:18.2.1-341.g4e3182a6.el8.x86_64 33/161 2024-01-24T14:16:31.079 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 29/161 2024-01-24T14:16:31.213 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 34/161 2024-01-24T14:16:31.365 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 30/161 2024-01-24T14:16:31.579 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 22/161 2024-01-24T14:16:31.586 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-ceph-common-2:18.2.1-341.g4e3182a6.el8.x 31/161 2024-01-24T14:16:31.743 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-ceph-argparse-2:18.2.1-341.g4e3182a6.el8 32/161 2024-01-24T14:16:31.746 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 23/161 2024-01-24T14:16:32.008 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cephfs-2:18.2.1-341.g4e3182a6.el8.x86_64 33/161 2024-01-24T14:16:32.416 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 23/161 2024-01-24T14:16:32.571 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : openblas-0.3.15-6.el8.x86_64 24/161 2024-01-24T14:16:32.953 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 34/161 2024-01-24T14:16:32.971 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 34/161 2024-01-24T14:16:33.320 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 35/161 2024-01-24T14:16:33.323 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 35/161 2024-01-24T14:16:33.330 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 24/161 2024-01-24T14:16:33.603 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 25/161 2024-01-24T14:16:33.634 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 36/161 2024-01-24T14:16:33.755 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 26/161 2024-01-24T14:16:33.872 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 37/161 2024-01-24T14:16:33.944 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 27/161 2024-01-24T14:16:34.220 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 28/161 2024-01-24T14:16:34.278 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 36/161 2024-01-24T14:16:34.531 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 29/161 2024-01-24T14:16:34.625 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 37/161 2024-01-24T14:16:34.824 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 30/161 2024-01-24T14:16:34.862 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 38/161 2024-01-24T14:16:34.994 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-ceph-common-2:18.2.1-341.g4e3182a6.el8.x 31/161 2024-01-24T14:16:35.115 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rsa-4.9-2.el8.noarch 39/161 2024-01-24T14:16:35.167 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-ceph-argparse-2:18.2.1-341.g4e3182a6.el8 32/161 2024-01-24T14:16:35.356 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 40/161 2024-01-24T14:16:35.412 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cephfs-2:18.2.1-341.g4e3182a6.el8.x86_64 33/161 2024-01-24T14:16:35.601 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rbd-2:18.2.1-341.g4e3182a6.el8.x86_64 41/161 2024-01-24T14:16:35.615 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 37/161 2024-01-24T14:16:35.760 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librados-devel-2:18.2.1-341.g4e3182a6.el8.x86_64 42/161 2024-01-24T14:16:35.930 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 38/161 2024-01-24T14:16:36.021 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : libmount-2.32.1-43.el8.x86_64 43/161 2024-01-24T14:16:36.097 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libmount-2.32.1-43.el8.x86_64 43/161 2024-01-24T14:16:36.182 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rsa-4.9-2.el8.noarch 39/161 2024-01-24T14:16:36.320 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : libfdisk-2.32.1-43.el8.x86_64 44/161 2024-01-24T14:16:36.423 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libfdisk-2.32.1-43.el8.x86_64 44/161 2024-01-24T14:16:36.431 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 40/161 2024-01-24T14:16:36.468 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 34/161 2024-01-24T14:16:36.610 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rbd-2:18.2.1-341.g4e3182a6.el8.x86_64 41/161 2024-01-24T14:16:36.641 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libuuid-devel-2.32.1-43.el8.x86_64 45/161 2024-01-24T14:16:36.819 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librados-devel-2:18.2.1-341.g4e3182a6.el8.x86_64 42/161 2024-01-24T14:16:36.822 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 35/161 2024-01-24T14:16:36.885 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 46/161 2024-01-24T14:16:37.056 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : libmount-2.32.1-43.el8.x86_64 43/161 2024-01-24T14:16:37.112 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : libsmartcols-2.32.1-43.el8.x86_64 47/161 2024-01-24T14:16:37.139 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 36/161 2024-01-24T14:16:37.166 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libmount-2.32.1-43.el8.x86_64 43/161 2024-01-24T14:16:37.312 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 37/161 2024-01-24T14:16:37.371 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : libfdisk-2.32.1-43.el8.x86_64 44/161 2024-01-24T14:16:37.424 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libfdisk-2.32.1-43.el8.x86_64 44/161 2024-01-24T14:16:37.567 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libsmartcols-2.32.1-43.el8.x86_64 47/161 2024-01-24T14:16:37.708 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libuuid-devel-2.32.1-43.el8.x86_64 45/161 2024-01-24T14:16:37.896 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : util-linux-2.32.1-43.el8.x86_64 48/161 2024-01-24T14:16:37.962 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 46/161 2024-01-24T14:16:37.988 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: util-linux-2.32.1-43.el8.x86_64 48/161 2024-01-24T14:16:38.146 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : libsmartcols-2.32.1-43.el8.x86_64 47/161 2024-01-24T14:16:38.262 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : automake-1.16.1-8.el8.noarch 49/161 2024-01-24T14:16:38.462 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : uuid-1.6.2-43.el8.x86_64 50/161 2024-01-24T14:16:38.531 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 50/161 2024-01-24T14:16:38.595 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libsmartcols-2.32.1-43.el8.x86_64 47/161 2024-01-24T14:16:38.712 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 51/161 2024-01-24T14:16:38.788 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 51/161 2024-01-24T14:16:38.905 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : util-linux-2.32.1-43.el8.x86_64 48/161 2024-01-24T14:16:38.923 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: util-linux-2.32.1-43.el8.x86_64 48/161 2024-01-24T14:16:39.003 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : jq-1.6-7.el8.x86_64 52/161 2024-01-24T14:16:39.166 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : fuse-2.9.7-17.el8.x86_64 49/161 2024-01-24T14:16:39.210 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 53/161 2024-01-24T14:16:39.288 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 37/161 2024-01-24T14:16:39.358 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 50/161 2024-01-24T14:16:39.360 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 54/161 2024-01-24T14:16:39.566 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 51/161 2024-01-24T14:16:39.597 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 55/161 2024-01-24T14:16:39.620 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 38/161 2024-01-24T14:16:39.771 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 52/161 2024-01-24T14:16:39.824 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : fuse-2.9.7-17.el8.x86_64 56/161 2024-01-24T14:16:39.865 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rsa-4.9-2.el8.noarch 39/161 2024-01-24T14:16:39.966 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 53/161 2024-01-24T14:16:39.993 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 57/161 2024-01-24T14:16:40.085 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 40/161 2024-01-24T14:16:40.200 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 58/161 2024-01-24T14:16:40.271 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 54/161 2024-01-24T14:16:40.303 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rbd-2:18.2.1-341.g4e3182a6.el8.x86_64 41/161 2024-01-24T14:16:40.420 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 59/161 2024-01-24T14:16:40.462 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librados-devel-2:18.2.1-341.g4e3182a6.el8.x86_64 42/161 2024-01-24T14:16:40.541 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 55/161 2024-01-24T14:16:40.657 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 60/161 2024-01-24T14:16:40.707 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : libmount-2.32.1-43.el8.x86_64 43/161 2024-01-24T14:16:40.767 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 56/161 2024-01-24T14:16:40.786 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libmount-2.32.1-43.el8.x86_64 43/161 2024-01-24T14:16:40.966 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 57/161 2024-01-24T14:16:40.997 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 61/161 2024-01-24T14:16:41.028 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : libfdisk-2.32.1-43.el8.x86_64 44/161 2024-01-24T14:16:41.087 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libfdisk-2.32.1-43.el8.x86_64 44/161 2024-01-24T14:16:41.189 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 58/161 2024-01-24T14:16:41.305 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libuuid-devel-2.32.1-43.el8.x86_64 45/161 2024-01-24T14:16:41.307 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 62/161 2024-01-24T14:16:41.523 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 46/161 2024-01-24T14:16:41.533 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 63/161 2024-01-24T14:16:41.568 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 59/161 2024-01-24T14:16:41.723 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 64/161 2024-01-24T14:16:41.725 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : libsmartcols-2.32.1-43.el8.x86_64 47/161 2024-01-24T14:16:41.882 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 60/161 2024-01-24T14:16:41.921 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 65/161 2024-01-24T14:16:42.105 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 61/161 2024-01-24T14:16:42.218 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libsmartcols-2.32.1-43.el8.x86_64 47/161 2024-01-24T14:16:42.268 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 66/161 2024-01-24T14:16:42.332 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 62/161 2024-01-24T14:16:42.512 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : util-linux-2.32.1-43.el8.x86_64 48/161 2024-01-24T14:16:42.533 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: util-linux-2.32.1-43.el8.x86_64 48/161 2024-01-24T14:16:42.579 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 67/161 2024-01-24T14:16:42.633 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 63/161 2024-01-24T14:16:42.725 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : fuse-2.9.7-17.el8.x86_64 49/161 2024-01-24T14:16:42.798 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 68/161 2024-01-24T14:16:42.947 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 50/161 2024-01-24T14:16:42.986 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 64/161 2024-01-24T14:16:43.092 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 69/161 2024-01-24T14:16:43.127 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 51/161 2024-01-24T14:16:43.228 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 65/161 2024-01-24T14:16:43.337 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 52/161 2024-01-24T14:16:43.374 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 70/161 2024-01-24T14:16:43.392 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 66/161 2024-01-24T14:16:43.526 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 53/161 2024-01-24T14:16:43.624 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 67/161 2024-01-24T14:16:43.744 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 71/161 2024-01-24T14:16:43.834 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 68/161 2024-01-24T14:16:43.848 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 54/161 2024-01-24T14:16:43.994 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 72/161 2024-01-24T14:16:44.114 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 55/161 2024-01-24T14:16:44.181 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : krb5-devel-1.18.2-26.el8_9.x86_64 69/161 2024-01-24T14:16:44.206 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 73/161 2024-01-24T14:16:44.334 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 56/161 2024-01-24T14:16:44.420 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : automake-1.16.1-8.el8.noarch 70/161 2024-01-24T14:16:44.439 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 74/161 2024-01-24T14:16:44.531 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 57/161 2024-01-24T14:16:44.595 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : uuid-1.6.2-43.el8.x86_64 71/161 2024-01-24T14:16:44.627 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 75/161 2024-01-24T14:16:44.668 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 71/161 2024-01-24T14:16:44.756 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 58/161 2024-01-24T14:16:44.870 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 72/161 2024-01-24T14:16:44.904 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : krb5-devel-1.18.2-26.el8_9.x86_64 76/161 2024-01-24T14:16:44.941 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 72/161 2024-01-24T14:16:45.103 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 77/161 2024-01-24T14:16:45.150 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 59/161 2024-01-24T14:16:45.178 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : jq-1.6-7.el8.x86_64 73/161 2024-01-24T14:16:45.324 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 78/161 2024-01-24T14:16:45.351 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 74/161 2024-01-24T14:16:45.485 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 60/161 2024-01-24T14:16:45.568 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 75/161 2024-01-24T14:16:45.640 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 79/161 2024-01-24T14:16:45.701 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 61/161 2024-01-24T14:16:45.746 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 76/161 2024-01-24T14:16:45.884 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 80/161 2024-01-24T14:16:45.922 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 62/161 2024-01-24T14:16:45.944 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 77/161 2024-01-24T14:16:46.116 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 78/161 2024-01-24T14:16:46.132 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 81/161 2024-01-24T14:16:46.188 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 63/161 2024-01-24T14:16:46.297 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 82/161 2024-01-24T14:16:46.418 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 79/161 2024-01-24T14:16:46.544 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 83/161 2024-01-24T14:16:46.567 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 64/161 2024-01-24T14:16:46.693 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 80/161 2024-01-24T14:16:46.801 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 84/161 2024-01-24T14:16:46.813 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 65/161 2024-01-24T14:16:46.899 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 81/161 2024-01-24T14:16:47.007 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 66/161 2024-01-24T14:16:47.041 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 85/161 2024-01-24T14:16:47.073 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 82/161 2024-01-24T14:16:47.214 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 67/161 2024-01-24T14:16:47.239 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 86/161 2024-01-24T14:16:47.262 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 83/161 2024-01-24T14:16:47.399 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 87/161 2024-01-24T14:16:47.419 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 68/161 2024-01-24T14:16:47.522 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 84/161 2024-01-24T14:16:47.679 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 88/161 2024-01-24T14:16:47.773 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : krb5-devel-1.18.2-26.el8_9.x86_64 69/161 2024-01-24T14:16:47.785 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 85/161 2024-01-24T14:16:47.922 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 89/161 2024-01-24T14:16:47.948 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 86/161 2024-01-24T14:16:47.997 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : automake-1.16.1-8.el8.noarch 70/161 2024-01-24T14:16:48.124 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 90/161 2024-01-24T14:16:48.133 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 87/161 2024-01-24T14:16:48.152 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : uuid-1.6.2-43.el8.x86_64 71/161 2024-01-24T14:16:48.242 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 71/161 2024-01-24T14:16:48.341 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 88/161 2024-01-24T14:16:48.354 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-portend-2.6-1.el8.noarch 91/161 2024-01-24T14:16:48.419 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 72/161 2024-01-24T14:16:48.507 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 72/161 2024-01-24T14:16:48.581 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 89/161 2024-01-24T14:16:48.626 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 92/161 2024-01-24T14:16:48.696 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : jq-1.6-7.el8.x86_64 73/161 2024-01-24T14:16:48.733 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 90/161 2024-01-24T14:16:48.809 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 93/161 2024-01-24T14:16:48.894 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 74/161 2024-01-24T14:16:49.007 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-portend-2.6-1.el8.noarch 91/161 2024-01-24T14:16:49.051 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 94/161 2024-01-24T14:16:49.070 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 75/161 2024-01-24T14:16:49.271 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 76/161 2024-01-24T14:16:49.277 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 92/161 2024-01-24T14:16:49.437 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 77/161 2024-01-24T14:16:49.452 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 93/161 2024-01-24T14:16:49.627 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 78/161 2024-01-24T14:16:49.668 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 94/161 2024-01-24T14:16:49.907 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 79/161 2024-01-24T14:16:50.119 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 80/161 2024-01-24T14:16:50.154 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 95/161 2024-01-24T14:16:50.332 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 81/161 2024-01-24T14:16:50.488 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 96/161 2024-01-24T14:16:50.489 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 82/161 2024-01-24T14:16:50.682 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 83/161 2024-01-24T14:16:50.728 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 95/161 2024-01-24T14:16:50.742 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 97/161 2024-01-24T14:16:50.935 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 84/161 2024-01-24T14:16:51.078 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 98/161 2024-01-24T14:16:51.081 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 96/161 2024-01-24T14:16:51.192 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 85/161 2024-01-24T14:16:51.310 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 97/161 2024-01-24T14:16:51.327 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 99/161 2024-01-24T14:16:51.381 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 86/161 2024-01-24T14:16:51.534 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 100/161 2024-01-24T14:16:51.566 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 87/161 2024-01-24T14:16:51.642 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 98/161 2024-01-24T14:16:51.796 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 88/161 2024-01-24T14:16:51.936 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 99/161 2024-01-24T14:16:52.041 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 89/161 2024-01-24T14:16:52.144 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 100/161 2024-01-24T14:16:52.208 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 90/161 2024-01-24T14:16:52.359 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 101/161 2024-01-24T14:16:52.499 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-portend-2.6-1.el8.noarch 91/161 2024-01-24T14:16:52.534 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librgw2-2:18.2.1-341.g4e3182a6.el8.x86_64 102/161 2024-01-24T14:16:52.609 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librgw2-2:18.2.1-341.g4e3182a6.el8.x86_64 102/161 2024-01-24T14:16:52.769 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 92/161 2024-01-24T14:16:52.827 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rgw-2:18.2.1-341.g4e3182a6.el8.x86_64 103/161 2024-01-24T14:16:52.944 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 93/161 2024-01-24T14:16:52.952 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 101/161 2024-01-24T14:16:53.197 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 94/161 2024-01-24T14:16:53.260 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librgw2-2:18.2.1-341.g4e3182a6.el8.x86_64 102/161 2024-01-24T14:16:53.344 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librgw2-2:18.2.1-341.g4e3182a6.el8.x86_64 102/161 2024-01-24T14:16:53.527 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rgw-2:18.2.1-341.g4e3182a6.el8.x86_64 103/161 2024-01-24T14:16:54.398 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 95/161 2024-01-24T14:16:54.747 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 96/161 2024-01-24T14:16:54.981 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 97/161 2024-01-24T14:16:55.308 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 98/161 2024-01-24T14:16:55.613 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 99/161 2024-01-24T14:16:55.839 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 100/161 2024-01-24T14:16:56.680 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 101/161 2024-01-24T14:16:56.826 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librgw2-2:18.2.1-341.g4e3182a6.el8.x86_64 102/161 2024-01-24T14:16:56.902 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librgw2-2:18.2.1-341.g4e3182a6.el8.x86_64 102/161 2024-01-24T14:16:57.110 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rgw-2:18.2.1-341.g4e3182a6.el8.x86_64 103/161 2024-01-24T14:16:57.123 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-common-2:18.2.1-341.g4e3182a6.el8.x86_64 104/161 2024-01-24T14:16:57.237 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-common-2:18.2.1-341.g4e3182a6.el8.x86_64 104/161 2024-01-24T14:16:57.401 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-common-2:18.2.1-341.g4e3182a6.el8.x86_64 104/161 2024-01-24T14:16:57.485 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-common-2:18.2.1-341.g4e3182a6.el8.x86_64 104/161 2024-01-24T14:16:57.978 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-common-2:18.2.1-341.g4e3182a6.el8.x86_64 104/161 2024-01-24T14:16:58.065 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-common-2:18.2.1-341.g4e3182a6.el8.x86_64 104/161 2024-01-24T14:16:58.291 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-base-2:18.2.1-341.g4e3182a6.el8.x86_64 105/161 2024-01-24T14:16:58.369 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-base-2:18.2.1-341.g4e3182a6.el8.x86_64 105/161 2024-01-24T14:16:58.567 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-selinux-2:18.2.1-341.g4e3182a6.el8.x86_64 106/161 2024-01-24T14:16:59.292 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-base-2:18.2.1-341.g4e3182a6.el8.x86_64 105/161 2024-01-24T14:16:59.837 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-base-2:18.2.1-341.g4e3182a6.el8.x86_64 105/161 2024-01-24T14:17:00.068 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-selinux-2:18.2.1-341.g4e3182a6.el8.x86_64 106/161 2024-01-24T14:17:01.679 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-common-2:18.2.1-341.g4e3182a6.el8.x86_64 104/161 2024-01-24T14:17:01.918 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-common-2:18.2.1-341.g4e3182a6.el8.x86_64 104/161 2024-01-24T14:17:02.566 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-common-2:18.2.1-341.g4e3182a6.el8.x86_64 104/161 2024-01-24T14:17:02.869 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-base-2:18.2.1-341.g4e3182a6.el8.x86_64 105/161 2024-01-24T14:17:02.961 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-base-2:18.2.1-341.g4e3182a6.el8.x86_64 105/161 2024-01-24T14:17:03.151 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-selinux-2:18.2.1-341.g4e3182a6.el8.x86_64 106/161 2024-01-24T14:17:17.957 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-selinux-2:18.2.1-341.g4e3182a6.el8.x86_64 106/161 2024-01-24T14:17:17.958 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /sys 2024-01-24T14:17:17.958 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /proc 2024-01-24T14:17:17.958 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /dev 2024-01-24T14:17:17.958 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /run 2024-01-24T14:17:17.958 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /mnt 2024-01-24T14:17:17.958 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /var/tmp 2024-01-24T14:17:17.958 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /home 2024-01-24T14:17:17.958 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /tmp 2024-01-24T14:17:17.958 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /dev 2024-01-24T14:17:17.958 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-24T14:17:18.339 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-volume-2:18.2.1-341.g4e3182a6.el8.noarch 107/161 2024-01-24T14:17:19.642 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-selinux-2:18.2.1-341.g4e3182a6.el8.x86_64 106/161 2024-01-24T14:17:19.642 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /sys 2024-01-24T14:17:19.642 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /proc 2024-01-24T14:17:19.642 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /dev 2024-01-24T14:17:19.642 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /run 2024-01-24T14:17:19.642 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /mnt 2024-01-24T14:17:19.642 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /var/tmp 2024-01-24T14:17:19.642 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /home 2024-01-24T14:17:19.642 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /tmp 2024-01-24T14:17:19.642 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /dev 2024-01-24T14:17:19.643 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-24T14:17:19.906 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-volume-2:18.2.1-341.g4e3182a6.el8.noarch 107/161 2024-01-24T14:17:20.039 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-volume-2:18.2.1-341.g4e3182a6.el8.noarch 107/161 2024-01-24T14:17:20.113 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-osd-2:18.2.1-341.g4e3182a6.el8.x86_64 108/161 2024-01-24T14:17:20.333 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-osd-2:18.2.1-341.g4e3182a6.el8.x86_64 108/161 2024-01-24T14:17:20.530 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mds-2:18.2.1-341.g4e3182a6.el8.x86_64 109/161 2024-01-24T14:17:20.986 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mds-2:18.2.1-341.g4e3182a6.el8.x86_64 109/161 2024-01-24T14:17:21.139 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mon-2:18.2.1-341.g4e3182a6.el8.x86_64 110/161 2024-01-24T14:17:21.157 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mon-2:18.2.1-341.g4e3182a6.el8.x86_64 110/161 2024-01-24T14:17:21.605 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-volume-2:18.2.1-341.g4e3182a6.el8.noarch 107/161 2024-01-24T14:17:21.803 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-osd-2:18.2.1-341.g4e3182a6.el8.x86_64 108/161 2024-01-24T14:17:22.022 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-osd-2:18.2.1-341.g4e3182a6.el8.x86_64 108/161 2024-01-24T14:17:22.189 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mds-2:18.2.1-341.g4e3182a6.el8.x86_64 109/161 2024-01-24T14:17:22.645 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mds-2:18.2.1-341.g4e3182a6.el8.x86_64 109/161 2024-01-24T14:17:22.814 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mon-2:18.2.1-341.g4e3182a6.el8.x86_64 110/161 2024-01-24T14:17:22.831 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mon-2:18.2.1-341.g4e3182a6.el8.x86_64 110/161 2024-01-24T14:17:22.935 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: cephadm-2:18.2.1-341.g4e3182a6.el8.noarch 111/161 2024-01-24T14:17:23.090 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : cephadm-2:18.2.1-341.g4e3182a6.el8.noarch 111/161 2024-01-24T14:17:23.236 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-prometheus-alerts-2:18.2.1-341.g4e3182a6.el 112/161 2024-01-24T14:17:23.447 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-grafana-dashboards-2:18.2.1-341.g4e3182a6.e 113/161 2024-01-24T14:17:23.713 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-cephadm-2:18.2.1-341.g4e3182a6.el8.noar 114/161 2024-01-24T14:17:23.937 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.1-341.g4e3182a6.el8.noar 114/161 2024-01-24T14:17:24.204 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-dashboard-2:18.2.1-341.g4e3182a6.el8.no 115/161 2024-01-24T14:17:24.741 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: cephadm-2:18.2.1-341.g4e3182a6.el8.noarch 111/161 2024-01-24T14:17:24.922 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : cephadm-2:18.2.1-341.g4e3182a6.el8.noarch 111/161 2024-01-24T14:17:25.061 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-prometheus-alerts-2:18.2.1-341.g4e3182a6.el 112/161 2024-01-24T14:17:25.169 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.1-341.g4e3182a6.el8.no 115/161 2024-01-24T14:17:25.220 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-grafana-dashboards-2:18.2.1-341.g4e3182a6.e 113/161 2024-01-24T14:17:25.338 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.1-341.g4e31 116/161 2024-01-24T14:17:25.430 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.1-341.g4e31 116/161 2024-01-24T14:17:25.471 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-cephadm-2:18.2.1-341.g4e3182a6.el8.noar 114/161 2024-01-24T14:17:25.697 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.1-341.g4e3182a6.el8.noar 114/161 2024-01-24T14:17:25.749 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-selinux-2:18.2.1-341.g4e3182a6.el8.x86_64 106/161 2024-01-24T14:17:25.749 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /sys 2024-01-24T14:17:25.749 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /proc 2024-01-24T14:17:25.749 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /dev 2024-01-24T14:17:25.749 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /run 2024-01-24T14:17:25.749 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /mnt 2024-01-24T14:17:25.749 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /var/tmp 2024-01-24T14:17:25.749 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /home 2024-01-24T14:17:25.749 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /tmp 2024-01-24T14:17:25.749 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /dev 2024-01-24T14:17:25.750 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:17:25.823 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-modules-core-2:18.2.1-341.g4e3182a6.el8 117/161 2024-01-24T14:17:26.011 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-dashboard-2:18.2.1-341.g4e3182a6.el8.no 115/161 2024-01-24T14:17:26.020 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-2:18.2.1-341.g4e3182a6.el8.x86_64 118/161 2024-01-24T14:17:26.058 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-2:18.2.1-341.g4e3182a6.el8.x86_64 118/161 2024-01-24T14:17:26.114 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-volume-2:18.2.1-341.g4e3182a6.el8.noarch 107/161 2024-01-24T14:17:26.237 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-rook-2:18.2.1-341.g4e3182a6.el8.noarch 119/161 2024-01-24T14:17:26.249 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.1-341.g4e3182a6.el8.noarch 119/161 2024-01-24T14:17:26.976 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.1-341.g4e3182a6.el8.no 115/161 2024-01-24T14:17:27.171 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.1-341.g4e31 116/161 2024-01-24T14:17:27.263 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.1-341.g4e31 116/161 2024-01-24T14:17:27.512 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-2:18.2.1-341.g4e3182a6.el8.x86_64 120/161 2024-01-24T14:17:27.647 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-modules-core-2:18.2.1-341.g4e3182a6.el8 117/161 2024-01-24T14:17:27.761 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-radosgw-2:18.2.1-341.g4e3182a6.el8.x86_64 121/161 2024-01-24T14:17:27.796 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-radosgw-2:18.2.1-341.g4e3182a6.el8.x86_64 121/161 2024-01-24T14:17:27.844 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-2:18.2.1-341.g4e3182a6.el8.x86_64 118/161 2024-01-24T14:17:27.849 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-volume-2:18.2.1-341.g4e3182a6.el8.noarch 107/161 2024-01-24T14:17:27.882 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-2:18.2.1-341.g4e3182a6.el8.x86_64 118/161 2024-01-24T14:17:27.988 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-immutable-object-cache-2:18.2.1-341.g4e3182 122/161 2024-01-24T14:17:28.029 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-rook-2:18.2.1-341.g4e3182a6.el8.noarch 119/161 2024-01-24T14:17:28.031 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.1-341.g4e3182 122/161 2024-01-24T14:17:28.032 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-osd-2:18.2.1-341.g4e3182a6.el8.x86_64 108/161 2024-01-24T14:17:28.041 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.1-341.g4e3182a6.el8.noarch 119/161 2024-01-24T14:17:28.229 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : cephfs-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 123/161 2024-01-24T14:17:28.279 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-osd-2:18.2.1-341.g4e3182a6.el8.x86_64 108/161 2024-01-24T14:17:28.448 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mds-2:18.2.1-341.g4e3182a6.el8.x86_64 109/161 2024-01-24T14:17:28.577 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: cephfs-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 123/161 2024-01-24T14:17:28.787 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : rbd-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 124/161 2024-01-24T14:17:28.954 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mds-2:18.2.1-341.g4e3182a6.el8.x86_64 109/161 2024-01-24T14:17:29.139 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mon-2:18.2.1-341.g4e3182a6.el8.x86_64 110/161 2024-01-24T14:17:29.162 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mon-2:18.2.1-341.g4e3182a6.el8.x86_64 110/161 2024-01-24T14:17:29.311 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-2:18.2.1-341.g4e3182a6.el8.x86_64 120/161 2024-01-24T14:17:29.536 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-radosgw-2:18.2.1-341.g4e3182a6.el8.x86_64 121/161 2024-01-24T14:17:29.569 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-radosgw-2:18.2.1-341.g4e3182a6.el8.x86_64 121/161 2024-01-24T14:17:29.779 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-immutable-object-cache-2:18.2.1-341.g4e3182 122/161 2024-01-24T14:17:29.823 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.1-341.g4e3182 122/161 2024-01-24T14:17:29.988 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : cephfs-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 123/161 2024-01-24T14:17:30.335 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: cephfs-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 123/161 2024-01-24T14:17:30.554 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 124/161 2024-01-24T14:17:31.124 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: cephadm-2:18.2.1-341.g4e3182a6.el8.noarch 111/161 2024-01-24T14:17:31.327 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : cephadm-2:18.2.1-341.g4e3182a6.el8.noarch 111/161 2024-01-24T14:17:31.438 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-prometheus-alerts-2:18.2.1-341.g4e3182a6.el 112/161 2024-01-24T14:17:31.637 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-grafana-dashboards-2:18.2.1-341.g4e3182a6.e 113/161 2024-01-24T14:17:31.880 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-cephadm-2:18.2.1-341.g4e3182a6.el8.noar 114/161 2024-01-24T14:17:32.123 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.1-341.g4e3182a6.el8.noar 114/161 2024-01-24T14:17:32.407 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-dashboard-2:18.2.1-341.g4e3182a6.el8.no 115/161 2024-01-24T14:17:33.201 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: rbd-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 124/161 2024-01-24T14:17:33.486 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.1-341.g4e3182a6.el8.no 115/161 2024-01-24T14:17:33.671 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.1-341.g4e31 116/161 2024-01-24T14:17:33.780 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.1-341.g4e31 116/161 2024-01-24T14:17:34.143 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-modules-core-2:18.2.1-341.g4e3182a6.el8 117/161 2024-01-24T14:17:34.201 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-test-2:18.2.1-341.g4e3182a6.el8.x86_64 125/161 2024-01-24T14:17:34.307 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-2:18.2.1-341.g4e3182a6.el8.x86_64 118/161 2024-01-24T14:17:34.354 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-2:18.2.1-341.g4e3182a6.el8.x86_64 118/161 2024-01-24T14:17:34.499 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : openssl-devel-1:1.1.1k-12.el8_9.x86_64 126/161 2024-01-24T14:17:34.529 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-rook-2:18.2.1-341.g4e3182a6.el8.noarch 119/161 2024-01-24T14:17:34.545 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.1-341.g4e3182a6.el8.noarch 119/161 2024-01-24T14:17:34.776 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : elfutils-libelf-devel-0.189-3.el8.x86_64 127/161 2024-01-24T14:17:34.985 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: rbd-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 124/161 2024-01-24T14:17:35.162 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-fuse-2:18.2.1-341.g4e3182a6.el8.x86_64 128/161 2024-01-24T14:17:35.455 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-test-2:18.2.1-341.g4e3182a6.el8.x86_64 125/161 2024-01-24T14:17:35.512 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : postgresql-contrib-10.23-3.module+el8.9.0+20831+ 129/161 2024-01-24T14:17:35.736 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libtool-2.4.6-25.el8.x86_64 130/161 2024-01-24T14:17:35.749 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 130/161 2024-01-24T14:17:35.788 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : postgresql-contrib-10.23-3.module+el8.9.0+20831+ 126/161 2024-01-24T14:17:35.920 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-2:18.2.1-341.g4e3182a6.el8.x86_64 120/161 2024-01-24T14:17:35.989 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : util-linux-user-2.32.1-43.el8.x86_64 131/161 2024-01-24T14:17:36.028 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libtool-2.4.6-25.el8.x86_64 127/161 2024-01-24T14:17:36.191 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-radosgw-2:18.2.1-341.g4e3182a6.el8.x86_64 121/161 2024-01-24T14:17:36.191 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 132/161 2024-01-24T14:17:36.232 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-radosgw-2:18.2.1-341.g4e3182a6.el8.x86_64 121/161 2024-01-24T14:17:36.362 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libcephfs-devel-2:18.2.1-341.g4e3182a6.el8.x86_6 133/161 2024-01-24T14:17:36.438 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-immutable-object-cache-2:18.2.1-341.g4e3182 122/161 2024-01-24T14:17:36.488 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.1-341.g4e3182 122/161 2024-01-24T14:17:36.655 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : cephfs-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 123/161 2024-01-24T14:17:36.711 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 127/161 2024-01-24T14:17:37.041 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : openssl-devel-1:1.1.1k-12.el8_9.x86_64 128/161 2024-01-24T14:17:37.041 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: cephfs-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 123/161 2024-01-24T14:17:37.246 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : rbd-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 124/161 2024-01-24T14:17:37.377 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : elfutils-libelf-devel-0.189-3.el8.x86_64 129/161 2024-01-24T14:17:37.651 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-fuse-2:18.2.1-341.g4e3182a6.el8.x86_64 130/161 2024-01-24T14:17:37.888 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : util-linux-user-2.32.1-43.el8.x86_64 131/161 2024-01-24T14:17:38.082 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 132/161 2024-01-24T14:17:38.295 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcephfs-devel-2:18.2.1-341.g4e3182a6.el8.x86_6 133/161 2024-01-24T14:17:38.682 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: postgresql-server-10.23-3.module+el8.9.0+20831+7 134/161 2024-01-24T14:17:39.011 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : postgresql-server-10.23-3.module+el8.9.0+20831+7 134/161 2024-01-24T14:17:39.049 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: postgresql-server-10.23-3.module+el8.9.0+20831+7 134/161 2024-01-24T14:17:39.471 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : elfutils-libs-0.189-3.el8.x86_64 135/161 2024-01-24T14:17:39.851 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : iproute-6.2.0-5.el8_9.x86_64 136/161 2024-01-24T14:17:40.106 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : xfsdump-3.1.8-7.el8_9.x86_64 137/161 2024-01-24T14:17:40.327 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : cephfs-top-2:18.2.1-341.g4e3182a6.el8.noarch 138/161 2024-01-24T14:17:40.526 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : rbd-fuse-2:18.2.1-341.g4e3182a6.el8.x86_64 139/161 2024-01-24T14:17:40.551 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: postgresql-server-10.23-3.module+el8.9.0+20831+7 134/161 2024-01-24T14:17:40.820 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : rbd-nbd-2:18.2.1-341.g4e3182a6.el8.x86_64 140/161 2024-01-24T14:17:40.886 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : postgresql-server-10.23-3.module+el8.9.0+20831+7 134/161 2024-01-24T14:17:40.924 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: postgresql-server-10.23-3.module+el8.9.0+20831+7 134/161 2024-01-24T14:17:41.146 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : openssl-1:1.1.1k-12.el8_9.x86_64 141/161 2024-01-24T14:17:41.335 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : flex-2.6.1-9.el8.x86_64 142/161 2024-01-24T14:17:41.374 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : elfutils-libs-0.189-3.el8.x86_64 135/161 2024-01-24T14:17:41.412 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 142/161 2024-01-24T14:17:41.610 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : bison-3.0.4-10.el8.x86_64 143/161 2024-01-24T14:17:41.623 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 143/161 2024-01-24T14:17:41.750 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : iproute-6.2.0-5.el8_9.x86_64 136/161 2024-01-24T14:17:41.810 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : NetworkManager-initscripts-updown-1:1.40.16-13.e 144/161 2024-01-24T14:17:41.819 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-13.e 144/161 2024-01-24T14:17:42.012 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 145/161 2024-01-24T14:17:42.023 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : xfsdump-3.1.8-7.el8_9.x86_64 137/161 2024-01-24T14:17:42.153 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: rbd-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 124/161 2024-01-24T14:17:42.211 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : cephfs-top-2:18.2.1-341.g4e3182a6.el8.noarch 138/161 2024-01-24T14:17:42.213 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : elfutils-libs-0.186-1.el8.x86_64 146/161 2024-01-24T14:17:42.378 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : openssl-1:1.1.1k-6.el8_5.x86_64 147/161 2024-01-24T14:17:42.477 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-fuse-2:18.2.1-341.g4e3182a6.el8.x86_64 139/161 2024-01-24T14:17:42.572 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 148/161 2024-01-24T14:17:42.573 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 149/161 2024-01-24T14:17:42.655 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-test-2:18.2.1-341.g4e3182a6.el8.x86_64 125/161 2024-01-24T14:17:42.809 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-nbd-2:18.2.1-341.g4e3182a6.el8.x86_64 140/161 2024-01-24T14:17:42.863 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 149/161 2024-01-24T14:17:42.863 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 150/161 2024-01-24T14:17:42.998 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : postgresql-contrib-10.23-3.module+el8.9.0+20831+ 126/161 2024-01-24T14:17:43.114 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 150/161 2024-01-24T14:17:43.114 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 151/161 2024-01-24T14:17:43.120 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : openssl-1:1.1.1k-12.el8_9.x86_64 141/161 2024-01-24T14:17:43.170 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libtool-2.4.6-25.el8.x86_64 127/161 2024-01-24T14:17:43.285 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : NetworkManager-initscripts-updown-1:1.40.16-13.e 142/161 2024-01-24T14:17:43.296 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-13.e 142/161 2024-01-24T14:17:43.391 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 151/161 2024-01-24T14:17:43.516 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 143/161 2024-01-24T14:17:43.653 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 152/161 2024-01-24T14:17:43.710 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : flex-2.6.1-9.el8.x86_64 144/161 2024-01-24T14:17:43.791 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 144/161 2024-01-24T14:17:43.828 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : util-linux-user-2.32.1-35.el8.x86_64 153/161 2024-01-24T14:17:43.998 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 127/161 2024-01-24T14:17:44.060 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : bison-3.0.4-10.el8.x86_64 145/161 2024-01-24T14:17:44.075 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 145/161 2024-01-24T14:17:44.086 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : util-linux-2.32.1-35.el8.x86_64 154/161 2024-01-24T14:17:44.086 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : libfdisk-2.32.1-35.el8.x86_64 155/161 2024-01-24T14:17:44.287 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : elfutils-libs-0.186-1.el8.x86_64 146/161 2024-01-24T14:17:44.318 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : openssl-devel-1:1.1.1k-12.el8_9.x86_64 128/161 2024-01-24T14:17:44.362 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libfdisk-2.32.1-35.el8.x86_64 155/161 2024-01-24T14:17:44.362 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : libmount-2.32.1-35.el8.x86_64 156/161 2024-01-24T14:17:44.454 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : openssl-1:1.1.1k-6.el8_5.x86_64 147/161 2024-01-24T14:17:44.611 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : elfutils-libelf-devel-0.189-3.el8.x86_64 129/161 2024-01-24T14:17:44.628 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libmount-2.32.1-35.el8.x86_64 156/161 2024-01-24T14:17:44.629 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : libblkid-2.32.1-35.el8.x86_64 157/161 2024-01-24T14:17:44.655 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 148/161 2024-01-24T14:17:44.656 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 149/161 2024-01-24T14:17:44.812 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-fuse-2:18.2.1-341.g4e3182a6.el8.x86_64 130/161 2024-01-24T14:17:44.904 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libblkid-2.32.1-35.el8.x86_64 157/161 2024-01-24T14:17:44.904 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : libuuid-2.32.1-35.el8.x86_64 158/161 2024-01-24T14:17:44.921 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 149/161 2024-01-24T14:17:44.921 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 150/161 2024-01-24T14:17:45.067 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : util-linux-user-2.32.1-43.el8.x86_64 131/161 2024-01-24T14:17:45.154 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libuuid-2.32.1-35.el8.x86_64 158/161 2024-01-24T14:17:45.154 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : libsmartcols-2.32.1-35.el8.x86_64 159/161 2024-01-24T14:17:45.172 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 150/161 2024-01-24T14:17:45.172 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 151/161 2024-01-24T14:17:45.234 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 132/161 2024-01-24T14:17:45.412 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libsmartcols-2.32.1-35.el8.x86_64 159/161 2024-01-24T14:17:45.446 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libcephfs-devel-2:18.2.1-341.g4e3182a6.el8.x86_6 133/161 2024-01-24T14:17:45.449 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 151/161 2024-01-24T14:17:45.595 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : elfutils-libelf-0.186-1.el8.x86_64 160/161 2024-01-24T14:17:45.596 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : openssl-libs-1:1.1.1k-6.el8_5.x86_64 161/161 2024-01-24T14:17:45.645 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 152/161 2024-01-24T14:17:45.845 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : util-linux-user-2.32.1-35.el8.x86_64 153/161 2024-01-24T14:17:46.095 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : util-linux-2.32.1-35.el8.x86_64 154/161 2024-01-24T14:17:46.095 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : libfdisk-2.32.1-35.el8.x86_64 155/161 2024-01-24T14:17:46.320 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libfdisk-2.32.1-35.el8.x86_64 155/161 2024-01-24T14:17:46.321 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : libmount-2.32.1-35.el8.x86_64 156/161 2024-01-24T14:17:46.554 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: openssl-libs-1:1.1.1k-6.el8_5.x86_64 161/161 2024-01-24T14:17:46.554 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-2:18.2.1-341.g4e3182a6.el8.x86_64 1/161 2024-01-24T14:17:46.554 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-base-2:18.2.1-341.g4e3182a6.el8.x86_64 2/161 2024-01-24T14:17:46.554 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-common-2:18.2.1-341.g4e3182a6.el8.x86_64 3/161 2024-01-24T14:17:46.555 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-fuse-2:18.2.1-341.g4e3182a6.el8.x86_64 4/161 2024-01-24T14:17:46.555 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-immutable-object-cache-2:18.2.1-341.g4e3182 5/161 2024-01-24T14:17:46.555 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mds-2:18.2.1-341.g4e3182a6.el8.x86_64 6/161 2024-01-24T14:17:46.555 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-2:18.2.1-341.g4e3182a6.el8.x86_64 7/161 2024-01-24T14:17:46.555 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mon-2:18.2.1-341.g4e3182a6.el8.x86_64 8/161 2024-01-24T14:17:46.555 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-osd-2:18.2.1-341.g4e3182a6.el8.x86_64 9/161 2024-01-24T14:17:46.555 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-radosgw-2:18.2.1-341.g4e3182a6.el8.x86_64 10/161 2024-01-24T14:17:46.555 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-selinux-2:18.2.1-341.g4e3182a6.el8.x86_64 11/161 2024-01-24T14:17:46.555 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-test-2:18.2.1-341.g4e3182a6.el8.x86_64 12/161 2024-01-24T14:17:46.555 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : cephfs-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 13/161 2024-01-24T14:17:46.555 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libcephfs-devel-2:18.2.1-341.g4e3182a6.el8.x86_6 14/161 2024-01-24T14:17:46.555 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libcephfs2-2:18.2.1-341.g4e3182a6.el8.x86_64 15/161 2024-01-24T14:17:46.555 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libcephsqlite-2:18.2.1-341.g4e3182a6.el8.x86_64 16/161 2024-01-24T14:17:46.556 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librados-devel-2:18.2.1-341.g4e3182a6.el8.x86_64 17/161 2024-01-24T14:17:46.556 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libradosstriper1-2:18.2.1-341.g4e3182a6.el8.x86_ 18/161 2024-01-24T14:17:46.556 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librgw2-2:18.2.1-341.g4e3182a6.el8.x86_64 19/161 2024-01-24T14:17:46.556 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-ceph-argparse-2:18.2.1-341.g4e3182a6.el8 20/161 2024-01-24T14:17:46.556 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-ceph-common-2:18.2.1-341.g4e3182a6.el8.x 21/161 2024-01-24T14:17:46.556 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cephfs-2:18.2.1-341.g4e3182a6.el8.x86_64 22/161 2024-01-24T14:17:46.556 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rados-2:18.2.1-341.g4e3182a6.el8.x86_64 23/161 2024-01-24T14:17:46.556 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rbd-2:18.2.1-341.g4e3182a6.el8.x86_64 24/161 2024-01-24T14:17:46.556 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rgw-2:18.2.1-341.g4e3182a6.el8.x86_64 25/161 2024-01-24T14:17:46.556 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : rbd-fuse-2:18.2.1-341.g4e3182a6.el8.x86_64 26/161 2024-01-24T14:17:46.556 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : rbd-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 27/161 2024-01-24T14:17:46.556 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : rbd-nbd-2:18.2.1-341.g4e3182a6.el8.x86_64 28/161 2024-01-24T14:17:46.556 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-grafana-dashboards-2:18.2.1-341.g4e3182a6.e 29/161 2024-01-24T14:17:46.557 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-cephadm-2:18.2.1-341.g4e3182a6.el8.noar 30/161 2024-01-24T14:17:46.557 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-dashboard-2:18.2.1-341.g4e3182a6.el8.no 31/161 2024-01-24T14:17:46.557 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.1-341.g4e31 32/161 2024-01-24T14:17:46.557 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-modules-core-2:18.2.1-341.g4e3182a6.el8 33/161 2024-01-24T14:17:46.569 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-rook-2:18.2.1-341.g4e3182a6.el8.noarch 34/161 2024-01-24T14:17:46.569 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-prometheus-alerts-2:18.2.1-341.g4e3182a6.el 35/161 2024-01-24T14:17:46.569 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-volume-2:18.2.1-341.g4e3182a6.el8.noarch 36/161 2024-01-24T14:17:46.569 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : cephadm-2:18.2.1-341.g4e3182a6.el8.noarch 37/161 2024-01-24T14:17:46.569 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : cephfs-top-2:18.2.1-341.g4e3182a6.el8.noarch 38/161 2024-01-24T14:17:46.569 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/161 2024-01-24T14:17:46.570 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/161 2024-01-24T14:17:46.570 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/161 2024-01-24T14:17:46.570 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/161 2024-01-24T14:17:46.570 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/161 2024-01-24T14:17:46.570 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/161 2024-01-24T14:17:46.570 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/161 2024-01-24T14:17:46.570 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/161 2024-01-24T14:17:46.570 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/161 2024-01-24T14:17:46.570 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/161 2024-01-24T14:17:46.570 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/161 2024-01-24T14:17:46.570 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/161 2024-01-24T14:17:46.570 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/161 2024-01-24T14:17:46.570 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/161 2024-01-24T14:17:46.571 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/161 2024-01-24T14:17:46.571 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/161 2024-01-24T14:17:46.571 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/161 2024-01-24T14:17:46.571 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/161 2024-01-24T14:17:46.571 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/161 2024-01-24T14:17:46.571 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/161 2024-01-24T14:17:46.571 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/161 2024-01-24T14:17:46.571 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/161 2024-01-24T14:17:46.571 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/161 2024-01-24T14:17:46.571 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/161 2024-01-24T14:17:46.571 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/161 2024-01-24T14:17:46.571 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/161 2024-01-24T14:17:46.571 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/161 2024-01-24T14:17:46.572 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/161 2024-01-24T14:17:46.572 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/161 2024-01-24T14:17:46.572 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/161 2024-01-24T14:17:46.572 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/161 2024-01-24T14:17:46.572 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/161 2024-01-24T14:17:46.572 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/161 2024-01-24T14:17:46.572 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 72/161 2024-01-24T14:17:46.572 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 73/161 2024-01-24T14:17:46.572 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 74/161 2024-01-24T14:17:46.572 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 75/161 2024-01-24T14:17:46.572 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 76/161 2024-01-24T14:17:46.572 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 77/161 2024-01-24T14:17:46.573 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 78/161 2024-01-24T14:17:46.573 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 79/161 2024-01-24T14:17:46.573 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/161 2024-01-24T14:17:46.573 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 81/161 2024-01-24T14:17:46.573 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 82/161 2024-01-24T14:17:46.573 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 83/161 2024-01-24T14:17:46.573 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 84/161 2024-01-24T14:17:46.573 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 85/161 2024-01-24T14:17:46.573 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 86/161 2024-01-24T14:17:46.573 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 87/161 2024-01-24T14:17:46.573 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 88/161 2024-01-24T14:17:46.573 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 89/161 2024-01-24T14:17:46.574 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libuuid-devel-2.32.1-43.el8.x86_64 90/161 2024-01-24T14:17:46.574 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 91/161 2024-01-24T14:17:46.574 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 92/161 2024-01-24T14:17:46.574 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 93/161 2024-01-24T14:17:46.574 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : elfutils-libelf-devel-0.189-3.el8.x86_64 94/161 2024-01-24T14:17:46.574 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 95/161 2024-01-24T14:17:46.574 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : krb5-devel-1.18.2-26.el8_9.x86_64 96/161 2024-01-24T14:17:46.574 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openssl-devel-1:1.1.1k-12.el8_9.x86_64 97/161 2024-01-24T14:17:46.574 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-13.e 98/161 2024-01-24T14:17:46.574 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 99/161 2024-01-24T14:17:46.574 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 100/161 2024-01-24T14:17:46.574 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 101/161 2024-01-24T14:17:46.574 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 102/161 2024-01-24T14:17:46.574 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 103/161 2024-01-24T14:17:46.575 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 104/161 2024-01-24T14:17:46.575 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : bison-3.0.4-10.el8.x86_64 105/161 2024-01-24T14:17:46.575 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : flex-2.6.1-9.el8.x86_64 106/161 2024-01-24T14:17:46.575 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 107/161 2024-01-24T14:17:46.575 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 108/161 2024-01-24T14:17:46.575 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 109/161 2024-01-24T14:17:46.575 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 110/161 2024-01-24T14:17:46.575 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 111/161 2024-01-24T14:17:46.575 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 112/161 2024-01-24T14:17:46.575 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 113/161 2024-01-24T14:17:46.575 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 114/161 2024-01-24T14:17:46.575 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 115/161 2024-01-24T14:17:46.575 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 116/161 2024-01-24T14:17:46.575 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 117/161 2024-01-24T14:17:46.576 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 118/161 2024-01-24T14:17:46.576 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 119/161 2024-01-24T14:17:46.576 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 120/161 2024-01-24T14:17:46.576 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 121/161 2024-01-24T14:17:46.576 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : automake-1.16.1-8.el8.noarch 122/161 2024-01-24T14:17:46.576 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libpq-13.11-1.el8.x86_64 123/161 2024-01-24T14:17:46.576 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : jq-1.6-7.el8.x86_64 124/161 2024-01-24T14:17:46.576 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 125/161 2024-01-24T14:17:46.576 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : postgresql-10.23-3.module+el8.9.0+20831+78489779 126/161 2024-01-24T14:17:46.576 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : postgresql-contrib-10.23-3.module+el8.9.0+20831+ 127/161 2024-01-24T14:17:46.576 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : postgresql-server-10.23-3.module+el8.9.0+20831+7 128/161 2024-01-24T14:17:46.576 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 129/161 2024-01-24T14:17:46.576 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librados2-2:18.2.1-341.g4e3182a6.el8.x86_64 130/161 2024-01-24T14:17:46.577 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 131/161 2024-01-24T14:17:46.577 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librbd1-2:18.2.1-341.g4e3182a6.el8.x86_64 132/161 2024-01-24T14:17:46.577 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 133/161 2024-01-24T14:17:46.577 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : elfutils-libelf-0.189-3.el8.x86_64 134/161 2024-01-24T14:17:46.577 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : elfutils-libelf-0.186-1.el8.x86_64 135/161 2024-01-24T14:17:46.577 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : elfutils-libs-0.189-3.el8.x86_64 136/161 2024-01-24T14:17:46.577 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : elfutils-libs-0.186-1.el8.x86_64 137/161 2024-01-24T14:17:46.577 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libmount-2.32.1-43.el8.x86_64 138/161 2024-01-24T14:17:46.577 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libmount-2.32.1-35.el8.x86_64 139/161 2024-01-24T14:17:46.577 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libsmartcols-2.32.1-43.el8.x86_64 140/161 2024-01-24T14:17:46.577 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libsmartcols-2.32.1-35.el8.x86_64 141/161 2024-01-24T14:17:46.577 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libuuid-2.32.1-43.el8.x86_64 142/161 2024-01-24T14:17:46.578 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libuuid-2.32.1-35.el8.x86_64 143/161 2024-01-24T14:17:46.578 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : util-linux-2.32.1-43.el8.x86_64 144/161 2024-01-24T14:17:46.578 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : util-linux-2.32.1-35.el8.x86_64 145/161 2024-01-24T14:17:46.578 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : util-linux-user-2.32.1-43.el8.x86_64 146/161 2024-01-24T14:17:46.578 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : util-linux-user-2.32.1-35.el8.x86_64 147/161 2024-01-24T14:17:46.578 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : iproute-6.2.0-5.el8_9.x86_64 148/161 2024-01-24T14:17:46.578 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 149/161 2024-01-24T14:17:46.578 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libblkid-2.32.1-43.el8.x86_64 150/161 2024-01-24T14:17:46.578 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libblkid-2.32.1-35.el8.x86_64 151/161 2024-01-24T14:17:46.578 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libfdisk-2.32.1-43.el8.x86_64 152/161 2024-01-24T14:17:46.578 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libfdisk-2.32.1-35.el8.x86_64 153/161 2024-01-24T14:17:46.578 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 154/161 2024-01-24T14:17:46.578 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 155/161 2024-01-24T14:17:46.578 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openssl-1:1.1.1k-12.el8_9.x86_64 156/161 2024-01-24T14:17:46.578 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openssl-1:1.1.1k-6.el8_5.x86_64 157/161 2024-01-24T14:17:46.579 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openssl-libs-1:1.1.1k-12.el8_9.x86_64 158/161 2024-01-24T14:17:46.579 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openssl-libs-1:1.1.1k-6.el8_5.x86_64 159/161 2024-01-24T14:17:46.579 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : xfsdump-3.1.8-7.el8_9.x86_64 160/161 2024-01-24T14:17:46.579 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libmount-2.32.1-35.el8.x86_64 156/161 2024-01-24T14:17:46.579 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : libblkid-2.32.1-35.el8.x86_64 157/161 2024-01-24T14:17:46.787 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libblkid-2.32.1-35.el8.x86_64 157/161 2024-01-24T14:17:46.787 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : libuuid-2.32.1-35.el8.x86_64 158/161 2024-01-24T14:17:47.045 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libuuid-2.32.1-35.el8.x86_64 158/161 2024-01-24T14:17:47.045 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : libsmartcols-2.32.1-35.el8.x86_64 159/161 2024-01-24T14:17:47.295 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libsmartcols-2.32.1-35.el8.x86_64 159/161 2024-01-24T14:17:47.462 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : elfutils-libelf-0.186-1.el8.x86_64 160/161 2024-01-24T14:17:47.462 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : openssl-libs-1:1.1.1k-6.el8_5.x86_64 161/161 2024-01-24T14:17:47.959 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: postgresql-server-10.23-3.module+el8.9.0+20831+7 134/161 2024-01-24T14:17:48.253 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : postgresql-server-10.23-3.module+el8.9.0+20831+7 134/161 2024-01-24T14:17:48.294 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: postgresql-server-10.23-3.module+el8.9.0+20831+7 134/161 2024-01-24T14:17:48.343 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: openssl-libs-1:1.1.1k-6.el8_5.x86_64 161/161 2024-01-24T14:17:48.343 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-2:18.2.1-341.g4e3182a6.el8.x86_64 1/161 2024-01-24T14:17:48.343 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-base-2:18.2.1-341.g4e3182a6.el8.x86_64 2/161 2024-01-24T14:17:48.343 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-common-2:18.2.1-341.g4e3182a6.el8.x86_64 3/161 2024-01-24T14:17:48.343 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-fuse-2:18.2.1-341.g4e3182a6.el8.x86_64 4/161 2024-01-24T14:17:48.343 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-immutable-object-cache-2:18.2.1-341.g4e3182 5/161 2024-01-24T14:17:48.343 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mds-2:18.2.1-341.g4e3182a6.el8.x86_64 6/161 2024-01-24T14:17:48.344 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-2:18.2.1-341.g4e3182a6.el8.x86_64 7/161 2024-01-24T14:17:48.344 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mon-2:18.2.1-341.g4e3182a6.el8.x86_64 8/161 2024-01-24T14:17:48.344 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-osd-2:18.2.1-341.g4e3182a6.el8.x86_64 9/161 2024-01-24T14:17:48.344 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-radosgw-2:18.2.1-341.g4e3182a6.el8.x86_64 10/161 2024-01-24T14:17:48.344 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-selinux-2:18.2.1-341.g4e3182a6.el8.x86_64 11/161 2024-01-24T14:17:48.344 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-test-2:18.2.1-341.g4e3182a6.el8.x86_64 12/161 2024-01-24T14:17:48.344 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : cephfs-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 13/161 2024-01-24T14:17:48.344 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcephfs-devel-2:18.2.1-341.g4e3182a6.el8.x86_6 14/161 2024-01-24T14:17:48.344 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcephfs2-2:18.2.1-341.g4e3182a6.el8.x86_64 15/161 2024-01-24T14:17:48.344 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcephsqlite-2:18.2.1-341.g4e3182a6.el8.x86_64 16/161 2024-01-24T14:17:48.344 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados-devel-2:18.2.1-341.g4e3182a6.el8.x86_64 17/161 2024-01-24T14:17:48.344 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libradosstriper1-2:18.2.1-341.g4e3182a6.el8.x86_ 18/161 2024-01-24T14:17:48.344 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librgw2-2:18.2.1-341.g4e3182a6.el8.x86_64 19/161 2024-01-24T14:17:48.344 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-ceph-argparse-2:18.2.1-341.g4e3182a6.el8 20/161 2024-01-24T14:17:48.345 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-ceph-common-2:18.2.1-341.g4e3182a6.el8.x 21/161 2024-01-24T14:17:48.345 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cephfs-2:18.2.1-341.g4e3182a6.el8.x86_64 22/161 2024-01-24T14:17:48.345 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rados-2:18.2.1-341.g4e3182a6.el8.x86_64 23/161 2024-01-24T14:17:48.345 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rbd-2:18.2.1-341.g4e3182a6.el8.x86_64 24/161 2024-01-24T14:17:48.345 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rgw-2:18.2.1-341.g4e3182a6.el8.x86_64 25/161 2024-01-24T14:17:48.345 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-fuse-2:18.2.1-341.g4e3182a6.el8.x86_64 26/161 2024-01-24T14:17:48.345 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 27/161 2024-01-24T14:17:48.345 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-nbd-2:18.2.1-341.g4e3182a6.el8.x86_64 28/161 2024-01-24T14:17:48.345 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-grafana-dashboards-2:18.2.1-341.g4e3182a6.e 29/161 2024-01-24T14:17:48.345 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-cephadm-2:18.2.1-341.g4e3182a6.el8.noar 30/161 2024-01-24T14:17:48.345 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-dashboard-2:18.2.1-341.g4e3182a6.el8.no 31/161 2024-01-24T14:17:48.345 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.1-341.g4e31 32/161 2024-01-24T14:17:48.345 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-modules-core-2:18.2.1-341.g4e3182a6.el8 33/161 2024-01-24T14:17:48.346 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-rook-2:18.2.1-341.g4e3182a6.el8.noarch 34/161 2024-01-24T14:17:48.346 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-prometheus-alerts-2:18.2.1-341.g4e3182a6.el 35/161 2024-01-24T14:17:48.346 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-volume-2:18.2.1-341.g4e3182a6.el8.noarch 36/161 2024-01-24T14:17:48.346 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : cephadm-2:18.2.1-341.g4e3182a6.el8.noarch 37/161 2024-01-24T14:17:48.346 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : cephfs-top-2:18.2.1-341.g4e3182a6.el8.noarch 38/161 2024-01-24T14:17:48.346 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/161 2024-01-24T14:17:48.346 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/161 2024-01-24T14:17:48.346 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/161 2024-01-24T14:17:48.346 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/161 2024-01-24T14:17:48.346 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/161 2024-01-24T14:17:48.346 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/161 2024-01-24T14:17:48.346 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/161 2024-01-24T14:17:48.346 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/161 2024-01-24T14:17:48.346 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/161 2024-01-24T14:17:48.347 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/161 2024-01-24T14:17:48.347 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/161 2024-01-24T14:17:48.347 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/161 2024-01-24T14:17:48.347 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/161 2024-01-24T14:17:48.347 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/161 2024-01-24T14:17:48.347 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/161 2024-01-24T14:17:48.347 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/161 2024-01-24T14:17:48.347 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/161 2024-01-24T14:17:48.347 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/161 2024-01-24T14:17:48.347 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/161 2024-01-24T14:17:48.347 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/161 2024-01-24T14:17:48.347 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/161 2024-01-24T14:17:48.347 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/161 2024-01-24T14:17:48.348 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/161 2024-01-24T14:17:48.348 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/161 2024-01-24T14:17:48.348 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/161 2024-01-24T14:17:48.348 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/161 2024-01-24T14:17:48.348 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/161 2024-01-24T14:17:48.348 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/161 2024-01-24T14:17:48.348 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/161 2024-01-24T14:17:48.348 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/161 2024-01-24T14:17:48.348 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/161 2024-01-24T14:17:48.348 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/161 2024-01-24T14:17:48.348 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/161 2024-01-24T14:17:48.348 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 72/161 2024-01-24T14:17:48.348 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 73/161 2024-01-24T14:17:48.349 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 74/161 2024-01-24T14:17:48.349 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 75/161 2024-01-24T14:17:48.349 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 76/161 2024-01-24T14:17:48.349 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 77/161 2024-01-24T14:17:48.349 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/161 2024-01-24T14:17:48.349 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : flex-2.6.1-9.el8.x86_64 79/161 2024-01-24T14:17:48.349 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 80/161 2024-01-24T14:17:48.349 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 81/161 2024-01-24T14:17:48.349 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 82/161 2024-01-24T14:17:48.349 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 83/161 2024-01-24T14:17:48.349 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 84/161 2024-01-24T14:17:48.349 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 85/161 2024-01-24T14:17:48.349 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 86/161 2024-01-24T14:17:48.349 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 87/161 2024-01-24T14:17:48.350 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 88/161 2024-01-24T14:17:48.350 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 89/161 2024-01-24T14:17:48.350 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 90/161 2024-01-24T14:17:48.350 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 91/161 2024-01-24T14:17:48.350 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 92/161 2024-01-24T14:17:48.350 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 93/161 2024-01-24T14:17:48.350 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 94/161 2024-01-24T14:17:48.350 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : automake-1.16.1-8.el8.noarch 95/161 2024-01-24T14:17:48.350 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libpq-13.11-1.el8.x86_64 96/161 2024-01-24T14:17:48.350 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : jq-1.6-7.el8.x86_64 97/161 2024-01-24T14:17:48.350 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 98/161 2024-01-24T14:17:48.350 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : postgresql-10.23-3.module+el8.9.0+20831+78489779 99/161 2024-01-24T14:17:48.350 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : postgresql-contrib-10.23-3.module+el8.9.0+20831+ 100/161 2024-01-24T14:17:48.350 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : postgresql-server-10.23-3.module+el8.9.0+20831+7 101/161 2024-01-24T14:17:48.351 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 102/161 2024-01-24T14:17:48.351 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 103/161 2024-01-24T14:17:48.351 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 104/161 2024-01-24T14:17:48.351 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 105/161 2024-01-24T14:17:48.351 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 106/161 2024-01-24T14:17:48.351 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 107/161 2024-01-24T14:17:48.351 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 108/161 2024-01-24T14:17:48.351 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 109/161 2024-01-24T14:17:48.351 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 110/161 2024-01-24T14:17:48.351 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/161 2024-01-24T14:17:48.351 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 112/161 2024-01-24T14:17:48.351 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 113/161 2024-01-24T14:17:48.351 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 114/161 2024-01-24T14:17:48.351 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 115/161 2024-01-24T14:17:48.352 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 116/161 2024-01-24T14:17:48.352 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 117/161 2024-01-24T14:17:48.352 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 118/161 2024-01-24T14:17:48.352 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 119/161 2024-01-24T14:17:48.352 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 120/161 2024-01-24T14:17:48.352 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libuuid-devel-2.32.1-43.el8.x86_64 121/161 2024-01-24T14:17:48.352 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 122/161 2024-01-24T14:17:48.352 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 123/161 2024-01-24T14:17:48.352 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 124/161 2024-01-24T14:17:48.352 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : elfutils-libelf-devel-0.189-3.el8.x86_64 125/161 2024-01-24T14:17:48.352 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 126/161 2024-01-24T14:17:48.352 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : krb5-devel-1.18.2-26.el8_9.x86_64 127/161 2024-01-24T14:17:48.352 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openssl-devel-1:1.1.1k-12.el8_9.x86_64 128/161 2024-01-24T14:17:48.353 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-13.e 129/161 2024-01-24T14:17:48.353 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados2-2:18.2.1-341.g4e3182a6.el8.x86_64 130/161 2024-01-24T14:17:48.353 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 131/161 2024-01-24T14:17:48.353 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librbd1-2:18.2.1-341.g4e3182a6.el8.x86_64 132/161 2024-01-24T14:17:48.353 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 133/161 2024-01-24T14:17:48.353 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : elfutils-libelf-0.189-3.el8.x86_64 134/161 2024-01-24T14:17:48.353 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : elfutils-libelf-0.186-1.el8.x86_64 135/161 2024-01-24T14:17:48.353 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : elfutils-libs-0.189-3.el8.x86_64 136/161 2024-01-24T14:17:48.353 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : elfutils-libs-0.186-1.el8.x86_64 137/161 2024-01-24T14:17:48.353 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libmount-2.32.1-43.el8.x86_64 138/161 2024-01-24T14:17:48.353 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libmount-2.32.1-35.el8.x86_64 139/161 2024-01-24T14:17:48.353 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libsmartcols-2.32.1-43.el8.x86_64 140/161 2024-01-24T14:17:48.353 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libsmartcols-2.32.1-35.el8.x86_64 141/161 2024-01-24T14:17:48.353 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libuuid-2.32.1-43.el8.x86_64 142/161 2024-01-24T14:17:48.354 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libuuid-2.32.1-35.el8.x86_64 143/161 2024-01-24T14:17:48.354 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : util-linux-2.32.1-43.el8.x86_64 144/161 2024-01-24T14:17:48.354 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : util-linux-2.32.1-35.el8.x86_64 145/161 2024-01-24T14:17:48.354 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : util-linux-user-2.32.1-43.el8.x86_64 146/161 2024-01-24T14:17:48.354 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : util-linux-user-2.32.1-35.el8.x86_64 147/161 2024-01-24T14:17:48.354 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : iproute-6.2.0-5.el8_9.x86_64 148/161 2024-01-24T14:17:48.354 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 149/161 2024-01-24T14:17:48.354 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libblkid-2.32.1-43.el8.x86_64 150/161 2024-01-24T14:17:48.354 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libblkid-2.32.1-35.el8.x86_64 151/161 2024-01-24T14:17:48.354 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libfdisk-2.32.1-43.el8.x86_64 152/161 2024-01-24T14:17:48.355 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libfdisk-2.32.1-35.el8.x86_64 153/161 2024-01-24T14:17:48.355 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 154/161 2024-01-24T14:17:48.355 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 155/161 2024-01-24T14:17:48.355 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openssl-1:1.1.1k-12.el8_9.x86_64 156/161 2024-01-24T14:17:48.355 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openssl-1:1.1.1k-6.el8_5.x86_64 157/161 2024-01-24T14:17:48.355 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openssl-libs-1:1.1.1k-12.el8_9.x86_64 158/161 2024-01-24T14:17:48.355 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openssl-libs-1:1.1.1k-6.el8_5.x86_64 159/161 2024-01-24T14:17:48.355 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : xfsdump-3.1.8-7.el8_9.x86_64 160/161 2024-01-24T14:17:48.725 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : elfutils-libs-0.189-3.el8.x86_64 135/161 2024-01-24T14:17:49.090 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : iproute-6.2.0-5.el8_9.x86_64 136/161 2024-01-24T14:17:49.341 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : xfsdump-3.1.8-7.el8_9.x86_64 137/161 2024-01-24T14:17:49.521 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : cephfs-top-2:18.2.1-341.g4e3182a6.el8.noarch 138/161 2024-01-24T14:17:49.738 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : rbd-fuse-2:18.2.1-341.g4e3182a6.el8.x86_64 139/161 2024-01-24T14:17:50.002 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : rbd-nbd-2:18.2.1-341.g4e3182a6.el8.x86_64 140/161 2024-01-24T14:17:50.380 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : openssl-1:1.1.1k-12.el8_9.x86_64 141/161 2024-01-24T14:17:50.586 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : NetworkManager-initscripts-updown-1:1.40.16-13.e 142/161 2024-01-24T14:17:50.598 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-13.e 142/161 2024-01-24T14:17:50.778 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 143/161 2024-01-24T14:17:50.985 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : flex-2.6.1-9.el8.x86_64 144/161 2024-01-24T14:17:51.073 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 144/161 2024-01-24T14:17:51.303 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : bison-3.0.4-10.el8.x86_64 145/161 2024-01-24T14:17:51.318 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 145/161 2024-01-24T14:17:51.493 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : elfutils-libs-0.186-1.el8.x86_64 146/161 2024-01-24T14:17:51.705 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : openssl-1:1.1.1k-6.el8_5.x86_64 147/161 2024-01-24T14:17:51.865 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 148/161 2024-01-24T14:17:51.865 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 149/161 2024-01-24T14:17:52.148 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 149/161 2024-01-24T14:17:52.148 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 150/161 2024-01-24T14:17:52.382 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 150/161 2024-01-24T14:17:52.382 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 151/161 2024-01-24T14:17:52.668 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 151/161 2024-01-24T14:17:52.896 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 152/161 2024-01-24T14:17:53.057 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : util-linux-user-2.32.1-35.el8.x86_64 153/161 2024-01-24T14:17:53.304 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : util-linux-2.32.1-35.el8.x86_64 154/161 2024-01-24T14:17:53.304 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : libfdisk-2.32.1-35.el8.x86_64 155/161 2024-01-24T14:17:53.514 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 161/161 2024-01-24T14:17:53.514 INFO:teuthology.orchestra.run.smithi123.stdout:Installed products updated. 2024-01-24T14:17:53.538 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libfdisk-2.32.1-35.el8.x86_64 155/161 2024-01-24T14:17:53.539 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : libmount-2.32.1-35.el8.x86_64 156/161 2024-01-24T14:17:53.772 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libmount-2.32.1-35.el8.x86_64 156/161 2024-01-24T14:17:53.772 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : libblkid-2.32.1-35.el8.x86_64 157/161 2024-01-24T14:17:54.022 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libblkid-2.32.1-35.el8.x86_64 157/161 2024-01-24T14:17:54.022 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : libuuid-2.32.1-35.el8.x86_64 158/161 2024-01-24T14:17:54.238 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libuuid-2.32.1-35.el8.x86_64 158/161 2024-01-24T14:17:54.238 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : libsmartcols-2.32.1-35.el8.x86_64 159/161 2024-01-24T14:17:54.488 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libsmartcols-2.32.1-35.el8.x86_64 159/161 2024-01-24T14:17:54.656 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : elfutils-libelf-0.186-1.el8.x86_64 160/161 2024-01-24T14:17:54.656 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : openssl-libs-1:1.1.1k-6.el8_5.x86_64 161/161 2024-01-24T14:17:54.750 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-24T14:17:54.750 INFO:teuthology.orchestra.run.smithi123.stdout:Upgraded: 2024-01-24T14:17:54.750 INFO:teuthology.orchestra.run.smithi123.stdout: elfutils-libelf-0.189-3.el8.x86_64 2024-01-24T14:17:54.750 INFO:teuthology.orchestra.run.smithi123.stdout: elfutils-libs-0.189-3.el8.x86_64 2024-01-24T14:17:54.750 INFO:teuthology.orchestra.run.smithi123.stdout: iproute-6.2.0-5.el8_9.x86_64 2024-01-24T14:17:54.750 INFO:teuthology.orchestra.run.smithi123.stdout: libblkid-2.32.1-43.el8.x86_64 2024-01-24T14:17:54.750 INFO:teuthology.orchestra.run.smithi123.stdout: libfdisk-2.32.1-43.el8.x86_64 2024-01-24T14:17:54.750 INFO:teuthology.orchestra.run.smithi123.stdout: libmount-2.32.1-43.el8.x86_64 2024-01-24T14:17:54.750 INFO:teuthology.orchestra.run.smithi123.stdout: librados2-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:54.751 INFO:teuthology.orchestra.run.smithi123.stdout: librbd1-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:54.751 INFO:teuthology.orchestra.run.smithi123.stdout: libsmartcols-2.32.1-43.el8.x86_64 2024-01-24T14:17:54.751 INFO:teuthology.orchestra.run.smithi123.stdout: libuuid-2.32.1-43.el8.x86_64 2024-01-24T14:17:54.751 INFO:teuthology.orchestra.run.smithi123.stdout: openssl-1:1.1.1k-12.el8_9.x86_64 2024-01-24T14:17:54.751 INFO:teuthology.orchestra.run.smithi123.stdout: openssl-libs-1:1.1.1k-12.el8_9.x86_64 2024-01-24T14:17:54.751 INFO:teuthology.orchestra.run.smithi123.stdout: util-linux-2.32.1-43.el8.x86_64 2024-01-24T14:17:54.751 INFO:teuthology.orchestra.run.smithi123.stdout: util-linux-user-2.32.1-43.el8.x86_64 2024-01-24T14:17:54.751 INFO:teuthology.orchestra.run.smithi123.stdout: xfsdump-3.1.8-7.el8_9.x86_64 2024-01-24T14:17:54.751 INFO:teuthology.orchestra.run.smithi123.stdout: xfsprogs-5.0.0-12.el8.x86_64 2024-01-24T14:17:54.751 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-01-24T14:17:54.751 INFO:teuthology.orchestra.run.smithi123.stdout: NetworkManager-initscripts-updown-1:1.40.16-13.el8_9.noarch 2024-01-24T14:17:54.751 INFO:teuthology.orchestra.run.smithi123.stdout: automake-1.16.1-8.el8.noarch 2024-01-24T14:17:54.751 INFO:teuthology.orchestra.run.smithi123.stdout: bison-3.0.4-10.el8.x86_64 2024-01-24T14:17:54.751 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:54.752 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-base-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:54.752 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-common-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:54.752 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-fuse-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:54.752 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-grafana-dashboards-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T14:17:54.752 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-immutable-object-cache-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:54.752 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mds-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:54.752 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:54.752 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-cephadm-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T14:17:54.752 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-dashboard-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T14:17:54.752 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-diskprediction-local-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T14:17:54.752 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-modules-core-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T14:17:54.752 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-rook-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T14:17:54.752 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mon-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:54.753 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-osd-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:54.753 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-prometheus-alerts-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T14:17:54.753 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-radosgw-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:54.753 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-selinux-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:54.753 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-test-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:54.753 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-volume-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T14:17:54.753 INFO:teuthology.orchestra.run.smithi123.stdout: cephadm-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T14:17:54.753 INFO:teuthology.orchestra.run.smithi123.stdout: cephfs-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:54.753 INFO:teuthology.orchestra.run.smithi123.stdout: cephfs-top-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T14:17:54.753 INFO:teuthology.orchestra.run.smithi123.stdout: elfutils-libelf-devel-0.189-3.el8.x86_64 2024-01-24T14:17:54.753 INFO:teuthology.orchestra.run.smithi123.stdout: flex-2.6.1-9.el8.x86_64 2024-01-24T14:17:54.753 INFO:teuthology.orchestra.run.smithi123.stdout: fuse-2.9.7-17.el8.x86_64 2024-01-24T14:17:54.753 INFO:teuthology.orchestra.run.smithi123.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-01-24T14:17:54.754 INFO:teuthology.orchestra.run.smithi123.stdout: jq-1.6-7.el8.x86_64 2024-01-24T14:17:54.754 INFO:teuthology.orchestra.run.smithi123.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2024-01-24T14:17:54.754 INFO:teuthology.orchestra.run.smithi123.stdout: krb5-devel-1.18.2-26.el8_9.x86_64 2024-01-24T14:17:54.754 INFO:teuthology.orchestra.run.smithi123.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-01-24T14:17:54.754 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs-devel-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:54.754 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs2-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:54.754 INFO:teuthology.orchestra.run.smithi123.stdout: libcephsqlite-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:54.754 INFO:teuthology.orchestra.run.smithi123.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2024-01-24T14:17:54.754 INFO:teuthology.orchestra.run.smithi123.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-01-24T14:17:54.754 INFO:teuthology.orchestra.run.smithi123.stdout: liboath-2.6.2-3.el8.x86_64 2024-01-24T14:17:54.754 INFO:teuthology.orchestra.run.smithi123.stdout: libpq-13.11-1.el8.x86_64 2024-01-24T14:17:54.754 INFO:teuthology.orchestra.run.smithi123.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-01-24T14:17:54.754 INFO:teuthology.orchestra.run.smithi123.stdout: librabbitmq-0.9.0-4.el8.x86_64 2024-01-24T14:17:54.755 INFO:teuthology.orchestra.run.smithi123.stdout: librados-devel-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:54.755 INFO:teuthology.orchestra.run.smithi123.stdout: libradosstriper1-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:54.755 INFO:teuthology.orchestra.run.smithi123.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-01-24T14:17:54.755 INFO:teuthology.orchestra.run.smithi123.stdout: librgw2-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:54.755 INFO:teuthology.orchestra.run.smithi123.stdout: libselinux-devel-2.9-5.el8.x86_64 2024-01-24T14:17:54.755 INFO:teuthology.orchestra.run.smithi123.stdout: libsepol-devel-2.9-3.el8.x86_64 2024-01-24T14:17:54.755 INFO:teuthology.orchestra.run.smithi123.stdout: libtool-2.4.6-25.el8.x86_64 2024-01-24T14:17:54.755 INFO:teuthology.orchestra.run.smithi123.stdout: libunwind-1.3.1-3.el8.x86_64 2024-01-24T14:17:54.755 INFO:teuthology.orchestra.run.smithi123.stdout: libuuid-devel-2.32.1-43.el8.x86_64 2024-01-24T14:17:54.755 INFO:teuthology.orchestra.run.smithi123.stdout: libverto-devel-0.3.0-5.el8.x86_64 2024-01-24T14:17:54.755 INFO:teuthology.orchestra.run.smithi123.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2024-01-24T14:17:54.755 INFO:teuthology.orchestra.run.smithi123.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-01-24T14:17:54.755 INFO:teuthology.orchestra.run.smithi123.stdout: oniguruma-6.8.2-2.el8.x86_64 2024-01-24T14:17:54.755 INFO:teuthology.orchestra.run.smithi123.stdout: openblas-0.3.15-6.el8.x86_64 2024-01-24T14:17:54.756 INFO:teuthology.orchestra.run.smithi123.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-01-24T14:17:54.756 INFO:teuthology.orchestra.run.smithi123.stdout: openssl-devel-1:1.1.1k-12.el8_9.x86_64 2024-01-24T14:17:54.756 INFO:teuthology.orchestra.run.smithi123.stdout: pcre2-devel-10.32-2.el8.x86_64 2024-01-24T14:17:54.756 INFO:teuthology.orchestra.run.smithi123.stdout: pcre2-utf16-10.32-2.el8.x86_64 2024-01-24T14:17:54.756 INFO:teuthology.orchestra.run.smithi123.stdout: pcre2-utf32-10.32-2.el8.x86_64 2024-01-24T14:17:54.756 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-10.23-3.module+el8.9.0+20831+78489779.x86_64 2024-01-24T14:17:54.756 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-contrib-10.23-3.module+el8.9.0+20831+78489779.x86_64 2024-01-24T14:17:54.756 INFO:teuthology.orchestra.run.smithi123.stdout: postgresql-server-10.23-3.module+el8.9.0+20831+78489779.x86_64 2024-01-24T14:17:54.756 INFO:teuthology.orchestra.run.smithi123.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-01-24T14:17:54.756 INFO:teuthology.orchestra.run.smithi123.stdout: python3-babel-2.5.1-7.el8.noarch 2024-01-24T14:17:54.756 INFO:teuthology.orchestra.run.smithi123.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-01-24T14:17:54.756 INFO:teuthology.orchestra.run.smithi123.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-01-24T14:17:54.756 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-01-24T14:17:54.756 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-argparse-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:54.757 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-common-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:54.757 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cephfs-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:54.757 INFO:teuthology.orchestra.run.smithi123.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-01-24T14:17:54.757 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-01-24T14:17:54.757 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-01-24T14:17:54.757 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-01-24T14:17:54.757 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2024-01-24T14:17:54.757 INFO:teuthology.orchestra.run.smithi123.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-01-24T14:17:54.757 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-6.2-6.el8.noarch 2024-01-24T14:17:54.757 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-01-24T14:17:54.757 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-01-24T14:17:54.757 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-01-24T14:17:54.757 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-01-24T14:17:54.757 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-01-24T14:17:54.758 INFO:teuthology.orchestra.run.smithi123.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-01-24T14:17:54.758 INFO:teuthology.orchestra.run.smithi123.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-01-24T14:17:54.758 INFO:teuthology.orchestra.run.smithi123.stdout: python3-mako-1.0.6-14.el8.noarch 2024-01-24T14:17:54.758 INFO:teuthology.orchestra.run.smithi123.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-01-24T14:17:54.758 INFO:teuthology.orchestra.run.smithi123.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-01-24T14:17:54.758 INFO:teuthology.orchestra.run.smithi123.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-01-24T14:17:54.758 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-01-24T14:17:54.758 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-01-24T14:17:54.758 INFO:teuthology.orchestra.run.smithi123.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-01-24T14:17:54.758 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-01-24T14:17:54.758 INFO:teuthology.orchestra.run.smithi123.stdout: python3-portend-2.6-1.el8.noarch 2024-01-24T14:17:54.758 INFO:teuthology.orchestra.run.smithi123.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-01-24T14:17:54.758 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-01-24T14:17:54.758 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-01-24T14:17:54.759 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-01-24T14:17:54.759 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pycparser-2.14-14.el8.noarch 2024-01-24T14:17:54.759 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pytz-2017.2-11.el8.noarch 2024-01-24T14:17:54.759 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rados-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:54.759 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rbd-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:54.759 INFO:teuthology.orchestra.run.smithi123.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-01-24T14:17:54.759 INFO:teuthology.orchestra.run.smithi123.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-01-24T14:17:54.759 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rgw-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:54.759 INFO:teuthology.orchestra.run.smithi123.stdout: python3-routes-2.4.1-12.el8.noarch 2024-01-24T14:17:54.759 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rsa-4.9-2.el8.noarch 2024-01-24T14:17:54.759 INFO:teuthology.orchestra.run.smithi123.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-01-24T14:17:54.759 INFO:teuthology.orchestra.run.smithi123.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-01-24T14:17:54.759 INFO:teuthology.orchestra.run.smithi123.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-01-24T14:17:54.759 INFO:teuthology.orchestra.run.smithi123.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-01-24T14:17:54.760 INFO:teuthology.orchestra.run.smithi123.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-01-24T14:17:54.760 INFO:teuthology.orchestra.run.smithi123.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-01-24T14:17:54.760 INFO:teuthology.orchestra.run.smithi123.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-01-24T14:17:54.760 INFO:teuthology.orchestra.run.smithi123.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-01-24T14:17:54.760 INFO:teuthology.orchestra.run.smithi123.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-01-24T14:17:54.760 INFO:teuthology.orchestra.run.smithi123.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-01-24T14:17:54.760 INFO:teuthology.orchestra.run.smithi123.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-01-24T14:17:54.760 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-fuse-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:54.760 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:54.760 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-nbd-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:54.760 INFO:teuthology.orchestra.run.smithi123.stdout: socat-1.7.4.1-1.el8.x86_64 2024-01-24T14:17:54.760 INFO:teuthology.orchestra.run.smithi123.stdout: thrift-0.13.0-2.el8.x86_64 2024-01-24T14:17:54.760 INFO:teuthology.orchestra.run.smithi123.stdout: uuid-1.6.2-43.el8.x86_64 2024-01-24T14:17:54.760 INFO:teuthology.orchestra.run.smithi123.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2024-01-24T14:17:54.761 INFO:teuthology.orchestra.run.smithi123.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-01-24T14:17:54.761 INFO:teuthology.orchestra.run.smithi123.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2024-01-24T14:17:54.761 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-01-24T14:17:54.761 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-01-24T14:17:55.251 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 161/161 2024-01-24T14:17:55.252 INFO:teuthology.orchestra.run.smithi194.stdout:Installed products updated. 2024-01-24T14:17:55.399 DEBUG:teuthology.parallel:result is None 2024-01-24T14:17:55.603 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: openssl-libs-1:1.1.1k-6.el8_5.x86_64 161/161 2024-01-24T14:17:55.603 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-2:18.2.1-341.g4e3182a6.el8.x86_64 1/161 2024-01-24T14:17:55.603 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-base-2:18.2.1-341.g4e3182a6.el8.x86_64 2/161 2024-01-24T14:17:55.603 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-common-2:18.2.1-341.g4e3182a6.el8.x86_64 3/161 2024-01-24T14:17:55.603 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-fuse-2:18.2.1-341.g4e3182a6.el8.x86_64 4/161 2024-01-24T14:17:55.603 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-immutable-object-cache-2:18.2.1-341.g4e3182 5/161 2024-01-24T14:17:55.603 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mds-2:18.2.1-341.g4e3182a6.el8.x86_64 6/161 2024-01-24T14:17:55.603 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-2:18.2.1-341.g4e3182a6.el8.x86_64 7/161 2024-01-24T14:17:55.603 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mon-2:18.2.1-341.g4e3182a6.el8.x86_64 8/161 2024-01-24T14:17:55.603 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-osd-2:18.2.1-341.g4e3182a6.el8.x86_64 9/161 2024-01-24T14:17:55.603 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-radosgw-2:18.2.1-341.g4e3182a6.el8.x86_64 10/161 2024-01-24T14:17:55.603 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-selinux-2:18.2.1-341.g4e3182a6.el8.x86_64 11/161 2024-01-24T14:17:55.604 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-test-2:18.2.1-341.g4e3182a6.el8.x86_64 12/161 2024-01-24T14:17:55.604 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : cephfs-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 13/161 2024-01-24T14:17:55.604 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libcephfs-devel-2:18.2.1-341.g4e3182a6.el8.x86_6 14/161 2024-01-24T14:17:55.604 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libcephfs2-2:18.2.1-341.g4e3182a6.el8.x86_64 15/161 2024-01-24T14:17:55.604 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libcephsqlite-2:18.2.1-341.g4e3182a6.el8.x86_64 16/161 2024-01-24T14:17:55.604 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librados-devel-2:18.2.1-341.g4e3182a6.el8.x86_64 17/161 2024-01-24T14:17:55.604 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libradosstriper1-2:18.2.1-341.g4e3182a6.el8.x86_ 18/161 2024-01-24T14:17:55.604 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librgw2-2:18.2.1-341.g4e3182a6.el8.x86_64 19/161 2024-01-24T14:17:55.604 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-ceph-argparse-2:18.2.1-341.g4e3182a6.el8 20/161 2024-01-24T14:17:55.604 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-ceph-common-2:18.2.1-341.g4e3182a6.el8.x 21/161 2024-01-24T14:17:55.604 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cephfs-2:18.2.1-341.g4e3182a6.el8.x86_64 22/161 2024-01-24T14:17:55.604 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rados-2:18.2.1-341.g4e3182a6.el8.x86_64 23/161 2024-01-24T14:17:55.604 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rbd-2:18.2.1-341.g4e3182a6.el8.x86_64 24/161 2024-01-24T14:17:55.604 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rgw-2:18.2.1-341.g4e3182a6.el8.x86_64 25/161 2024-01-24T14:17:55.605 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : rbd-fuse-2:18.2.1-341.g4e3182a6.el8.x86_64 26/161 2024-01-24T14:17:55.605 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : rbd-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 27/161 2024-01-24T14:17:55.605 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : rbd-nbd-2:18.2.1-341.g4e3182a6.el8.x86_64 28/161 2024-01-24T14:17:55.605 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-grafana-dashboards-2:18.2.1-341.g4e3182a6.e 29/161 2024-01-24T14:17:55.605 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-cephadm-2:18.2.1-341.g4e3182a6.el8.noar 30/161 2024-01-24T14:17:55.616 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-dashboard-2:18.2.1-341.g4e3182a6.el8.no 31/161 2024-01-24T14:17:55.616 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.1-341.g4e31 32/161 2024-01-24T14:17:55.616 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-modules-core-2:18.2.1-341.g4e3182a6.el8 33/161 2024-01-24T14:17:55.617 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-rook-2:18.2.1-341.g4e3182a6.el8.noarch 34/161 2024-01-24T14:17:55.617 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-prometheus-alerts-2:18.2.1-341.g4e3182a6.el 35/161 2024-01-24T14:17:55.617 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-volume-2:18.2.1-341.g4e3182a6.el8.noarch 36/161 2024-01-24T14:17:55.617 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : cephadm-2:18.2.1-341.g4e3182a6.el8.noarch 37/161 2024-01-24T14:17:55.617 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : cephfs-top-2:18.2.1-341.g4e3182a6.el8.noarch 38/161 2024-01-24T14:17:55.617 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/161 2024-01-24T14:17:55.617 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/161 2024-01-24T14:17:55.617 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/161 2024-01-24T14:17:55.617 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/161 2024-01-24T14:17:55.617 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/161 2024-01-24T14:17:55.617 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/161 2024-01-24T14:17:55.617 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/161 2024-01-24T14:17:55.617 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/161 2024-01-24T14:17:55.617 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/161 2024-01-24T14:17:55.618 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/161 2024-01-24T14:17:55.618 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/161 2024-01-24T14:17:55.618 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/161 2024-01-24T14:17:55.618 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/161 2024-01-24T14:17:55.618 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/161 2024-01-24T14:17:55.618 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/161 2024-01-24T14:17:55.618 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/161 2024-01-24T14:17:55.618 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/161 2024-01-24T14:17:55.618 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/161 2024-01-24T14:17:55.618 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/161 2024-01-24T14:17:55.618 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/161 2024-01-24T14:17:55.618 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/161 2024-01-24T14:17:55.618 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/161 2024-01-24T14:17:55.618 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/161 2024-01-24T14:17:55.618 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/161 2024-01-24T14:17:55.619 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/161 2024-01-24T14:17:55.619 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/161 2024-01-24T14:17:55.619 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/161 2024-01-24T14:17:55.619 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/161 2024-01-24T14:17:55.619 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/161 2024-01-24T14:17:55.619 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/161 2024-01-24T14:17:55.619 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/161 2024-01-24T14:17:55.619 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/161 2024-01-24T14:17:55.619 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/161 2024-01-24T14:17:55.619 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 72/161 2024-01-24T14:17:55.619 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 73/161 2024-01-24T14:17:55.619 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 74/161 2024-01-24T14:17:55.619 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 75/161 2024-01-24T14:17:55.619 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 76/161 2024-01-24T14:17:55.619 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 77/161 2024-01-24T14:17:55.620 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/161 2024-01-24T14:17:55.620 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : flex-2.6.1-9.el8.x86_64 79/161 2024-01-24T14:17:55.620 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 80/161 2024-01-24T14:17:55.620 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 81/161 2024-01-24T14:17:55.620 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 82/161 2024-01-24T14:17:55.620 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 83/161 2024-01-24T14:17:55.620 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 84/161 2024-01-24T14:17:55.620 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 85/161 2024-01-24T14:17:55.620 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 86/161 2024-01-24T14:17:55.620 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 87/161 2024-01-24T14:17:55.620 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 88/161 2024-01-24T14:17:55.620 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 89/161 2024-01-24T14:17:55.620 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 90/161 2024-01-24T14:17:55.621 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 91/161 2024-01-24T14:17:55.621 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 92/161 2024-01-24T14:17:55.621 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 93/161 2024-01-24T14:17:55.621 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 94/161 2024-01-24T14:17:55.621 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : automake-1.16.1-8.el8.noarch 95/161 2024-01-24T14:17:55.621 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libpq-13.11-1.el8.x86_64 96/161 2024-01-24T14:17:55.621 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : jq-1.6-7.el8.x86_64 97/161 2024-01-24T14:17:55.621 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 98/161 2024-01-24T14:17:55.621 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : postgresql-10.23-3.module+el8.9.0+20831+78489779 99/161 2024-01-24T14:17:55.621 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : postgresql-contrib-10.23-3.module+el8.9.0+20831+ 100/161 2024-01-24T14:17:55.621 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : postgresql-server-10.23-3.module+el8.9.0+20831+7 101/161 2024-01-24T14:17:55.621 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 102/161 2024-01-24T14:17:55.621 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 103/161 2024-01-24T14:17:55.621 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 104/161 2024-01-24T14:17:55.621 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 105/161 2024-01-24T14:17:55.622 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 106/161 2024-01-24T14:17:55.622 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 107/161 2024-01-24T14:17:55.622 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 108/161 2024-01-24T14:17:55.622 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 109/161 2024-01-24T14:17:55.622 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 110/161 2024-01-24T14:17:55.622 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/161 2024-01-24T14:17:55.622 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 112/161 2024-01-24T14:17:55.622 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 113/161 2024-01-24T14:17:55.622 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 114/161 2024-01-24T14:17:55.622 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 115/161 2024-01-24T14:17:55.622 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 116/161 2024-01-24T14:17:55.622 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 117/161 2024-01-24T14:17:55.622 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 118/161 2024-01-24T14:17:55.622 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 119/161 2024-01-24T14:17:55.622 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 120/161 2024-01-24T14:17:55.622 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libuuid-devel-2.32.1-43.el8.x86_64 121/161 2024-01-24T14:17:55.623 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 122/161 2024-01-24T14:17:55.623 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 123/161 2024-01-24T14:17:55.623 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 124/161 2024-01-24T14:17:55.623 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : elfutils-libelf-devel-0.189-3.el8.x86_64 125/161 2024-01-24T14:17:55.623 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 126/161 2024-01-24T14:17:55.623 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : krb5-devel-1.18.2-26.el8_9.x86_64 127/161 2024-01-24T14:17:55.623 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : openssl-devel-1:1.1.1k-12.el8_9.x86_64 128/161 2024-01-24T14:17:55.623 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-13.e 129/161 2024-01-24T14:17:55.623 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librados2-2:18.2.1-341.g4e3182a6.el8.x86_64 130/161 2024-01-24T14:17:55.623 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 131/161 2024-01-24T14:17:55.623 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librbd1-2:18.2.1-341.g4e3182a6.el8.x86_64 132/161 2024-01-24T14:17:55.623 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 133/161 2024-01-24T14:17:55.623 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : elfutils-libelf-0.189-3.el8.x86_64 134/161 2024-01-24T14:17:55.623 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : elfutils-libelf-0.186-1.el8.x86_64 135/161 2024-01-24T14:17:55.623 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : elfutils-libs-0.189-3.el8.x86_64 136/161 2024-01-24T14:17:55.623 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : elfutils-libs-0.186-1.el8.x86_64 137/161 2024-01-24T14:17:55.623 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libmount-2.32.1-43.el8.x86_64 138/161 2024-01-24T14:17:55.624 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libmount-2.32.1-35.el8.x86_64 139/161 2024-01-24T14:17:55.624 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libsmartcols-2.32.1-43.el8.x86_64 140/161 2024-01-24T14:17:55.624 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libsmartcols-2.32.1-35.el8.x86_64 141/161 2024-01-24T14:17:55.624 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libuuid-2.32.1-43.el8.x86_64 142/161 2024-01-24T14:17:55.624 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libuuid-2.32.1-35.el8.x86_64 143/161 2024-01-24T14:17:55.624 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : util-linux-2.32.1-43.el8.x86_64 144/161 2024-01-24T14:17:55.624 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : util-linux-2.32.1-35.el8.x86_64 145/161 2024-01-24T14:17:55.624 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : util-linux-user-2.32.1-43.el8.x86_64 146/161 2024-01-24T14:17:55.624 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : util-linux-user-2.32.1-35.el8.x86_64 147/161 2024-01-24T14:17:55.624 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : iproute-6.2.0-5.el8_9.x86_64 148/161 2024-01-24T14:17:55.624 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 149/161 2024-01-24T14:17:55.624 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libblkid-2.32.1-43.el8.x86_64 150/161 2024-01-24T14:17:55.624 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libblkid-2.32.1-35.el8.x86_64 151/161 2024-01-24T14:17:55.624 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libfdisk-2.32.1-43.el8.x86_64 152/161 2024-01-24T14:17:55.625 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libfdisk-2.32.1-35.el8.x86_64 153/161 2024-01-24T14:17:55.625 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 154/161 2024-01-24T14:17:55.625 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 155/161 2024-01-24T14:17:55.625 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : openssl-1:1.1.1k-12.el8_9.x86_64 156/161 2024-01-24T14:17:55.625 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : openssl-1:1.1.1k-6.el8_5.x86_64 157/161 2024-01-24T14:17:55.625 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : openssl-libs-1:1.1.1k-12.el8_9.x86_64 158/161 2024-01-24T14:17:55.625 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : openssl-libs-1:1.1.1k-6.el8_5.x86_64 159/161 2024-01-24T14:17:55.625 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : xfsdump-3.1.8-7.el8_9.x86_64 160/161 2024-01-24T14:17:56.271 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-24T14:17:56.272 INFO:teuthology.orchestra.run.smithi194.stdout:Upgraded: 2024-01-24T14:17:56.272 INFO:teuthology.orchestra.run.smithi194.stdout: elfutils-libelf-0.189-3.el8.x86_64 2024-01-24T14:17:56.272 INFO:teuthology.orchestra.run.smithi194.stdout: elfutils-libs-0.189-3.el8.x86_64 2024-01-24T14:17:56.272 INFO:teuthology.orchestra.run.smithi194.stdout: iproute-6.2.0-5.el8_9.x86_64 2024-01-24T14:17:56.272 INFO:teuthology.orchestra.run.smithi194.stdout: libblkid-2.32.1-43.el8.x86_64 2024-01-24T14:17:56.272 INFO:teuthology.orchestra.run.smithi194.stdout: libfdisk-2.32.1-43.el8.x86_64 2024-01-24T14:17:56.272 INFO:teuthology.orchestra.run.smithi194.stdout: libmount-2.32.1-43.el8.x86_64 2024-01-24T14:17:56.272 INFO:teuthology.orchestra.run.smithi194.stdout: librados2-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:56.272 INFO:teuthology.orchestra.run.smithi194.stdout: librbd1-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:56.272 INFO:teuthology.orchestra.run.smithi194.stdout: libsmartcols-2.32.1-43.el8.x86_64 2024-01-24T14:17:56.272 INFO:teuthology.orchestra.run.smithi194.stdout: libuuid-2.32.1-43.el8.x86_64 2024-01-24T14:17:56.272 INFO:teuthology.orchestra.run.smithi194.stdout: openssl-1:1.1.1k-12.el8_9.x86_64 2024-01-24T14:17:56.272 INFO:teuthology.orchestra.run.smithi194.stdout: openssl-libs-1:1.1.1k-12.el8_9.x86_64 2024-01-24T14:17:56.273 INFO:teuthology.orchestra.run.smithi194.stdout: util-linux-2.32.1-43.el8.x86_64 2024-01-24T14:17:56.273 INFO:teuthology.orchestra.run.smithi194.stdout: util-linux-user-2.32.1-43.el8.x86_64 2024-01-24T14:17:56.273 INFO:teuthology.orchestra.run.smithi194.stdout: xfsdump-3.1.8-7.el8_9.x86_64 2024-01-24T14:17:56.273 INFO:teuthology.orchestra.run.smithi194.stdout: xfsprogs-5.0.0-12.el8.x86_64 2024-01-24T14:17:56.273 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-01-24T14:17:56.273 INFO:teuthology.orchestra.run.smithi194.stdout: NetworkManager-initscripts-updown-1:1.40.16-13.el8_9.noarch 2024-01-24T14:17:56.273 INFO:teuthology.orchestra.run.smithi194.stdout: automake-1.16.1-8.el8.noarch 2024-01-24T14:17:56.273 INFO:teuthology.orchestra.run.smithi194.stdout: bison-3.0.4-10.el8.x86_64 2024-01-24T14:17:56.273 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:56.273 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-base-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:56.273 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-common-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:56.273 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-fuse-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:56.273 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-grafana-dashboards-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T14:17:56.273 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-immutable-object-cache-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:56.274 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mds-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:56.274 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:56.274 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-cephadm-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T14:17:56.274 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-dashboard-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T14:17:56.274 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-diskprediction-local-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T14:17:56.274 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-modules-core-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T14:17:56.274 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-rook-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T14:17:56.274 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mon-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:56.274 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-osd-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:56.274 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-prometheus-alerts-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T14:17:56.274 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-radosgw-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:56.274 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-selinux-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:56.274 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-test-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:56.274 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-volume-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T14:17:56.275 INFO:teuthology.orchestra.run.smithi194.stdout: cephadm-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T14:17:56.275 INFO:teuthology.orchestra.run.smithi194.stdout: cephfs-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:56.275 INFO:teuthology.orchestra.run.smithi194.stdout: cephfs-top-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T14:17:56.275 INFO:teuthology.orchestra.run.smithi194.stdout: elfutils-libelf-devel-0.189-3.el8.x86_64 2024-01-24T14:17:56.275 INFO:teuthology.orchestra.run.smithi194.stdout: flex-2.6.1-9.el8.x86_64 2024-01-24T14:17:56.275 INFO:teuthology.orchestra.run.smithi194.stdout: fuse-2.9.7-17.el8.x86_64 2024-01-24T14:17:56.275 INFO:teuthology.orchestra.run.smithi194.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-01-24T14:17:56.275 INFO:teuthology.orchestra.run.smithi194.stdout: jq-1.6-7.el8.x86_64 2024-01-24T14:17:56.275 INFO:teuthology.orchestra.run.smithi194.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2024-01-24T14:17:56.275 INFO:teuthology.orchestra.run.smithi194.stdout: krb5-devel-1.18.2-26.el8_9.x86_64 2024-01-24T14:17:56.275 INFO:teuthology.orchestra.run.smithi194.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-01-24T14:17:56.275 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs-devel-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:56.275 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs2-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:56.275 INFO:teuthology.orchestra.run.smithi194.stdout: libcephsqlite-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:56.276 INFO:teuthology.orchestra.run.smithi194.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2024-01-24T14:17:56.276 INFO:teuthology.orchestra.run.smithi194.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-01-24T14:17:56.276 INFO:teuthology.orchestra.run.smithi194.stdout: liboath-2.6.2-3.el8.x86_64 2024-01-24T14:17:56.276 INFO:teuthology.orchestra.run.smithi194.stdout: libpq-13.11-1.el8.x86_64 2024-01-24T14:17:56.276 INFO:teuthology.orchestra.run.smithi194.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-01-24T14:17:56.276 INFO:teuthology.orchestra.run.smithi194.stdout: librabbitmq-0.9.0-4.el8.x86_64 2024-01-24T14:17:56.276 INFO:teuthology.orchestra.run.smithi194.stdout: librados-devel-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:56.276 INFO:teuthology.orchestra.run.smithi194.stdout: libradosstriper1-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:56.276 INFO:teuthology.orchestra.run.smithi194.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-01-24T14:17:56.276 INFO:teuthology.orchestra.run.smithi194.stdout: librgw2-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:56.276 INFO:teuthology.orchestra.run.smithi194.stdout: libselinux-devel-2.9-5.el8.x86_64 2024-01-24T14:17:56.276 INFO:teuthology.orchestra.run.smithi194.stdout: libsepol-devel-2.9-3.el8.x86_64 2024-01-24T14:17:56.276 INFO:teuthology.orchestra.run.smithi194.stdout: libtool-2.4.6-25.el8.x86_64 2024-01-24T14:17:56.276 INFO:teuthology.orchestra.run.smithi194.stdout: libunwind-1.3.1-3.el8.x86_64 2024-01-24T14:17:56.277 INFO:teuthology.orchestra.run.smithi194.stdout: libuuid-devel-2.32.1-43.el8.x86_64 2024-01-24T14:17:56.277 INFO:teuthology.orchestra.run.smithi194.stdout: libverto-devel-0.3.0-5.el8.x86_64 2024-01-24T14:17:56.277 INFO:teuthology.orchestra.run.smithi194.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2024-01-24T14:17:56.277 INFO:teuthology.orchestra.run.smithi194.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-01-24T14:17:56.277 INFO:teuthology.orchestra.run.smithi194.stdout: oniguruma-6.8.2-2.el8.x86_64 2024-01-24T14:17:56.277 INFO:teuthology.orchestra.run.smithi194.stdout: openblas-0.3.15-6.el8.x86_64 2024-01-24T14:17:56.277 INFO:teuthology.orchestra.run.smithi194.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-01-24T14:17:56.277 INFO:teuthology.orchestra.run.smithi194.stdout: openssl-devel-1:1.1.1k-12.el8_9.x86_64 2024-01-24T14:17:56.277 INFO:teuthology.orchestra.run.smithi194.stdout: pcre2-devel-10.32-2.el8.x86_64 2024-01-24T14:17:56.277 INFO:teuthology.orchestra.run.smithi194.stdout: pcre2-utf16-10.32-2.el8.x86_64 2024-01-24T14:17:56.277 INFO:teuthology.orchestra.run.smithi194.stdout: pcre2-utf32-10.32-2.el8.x86_64 2024-01-24T14:17:56.277 INFO:teuthology.orchestra.run.smithi194.stdout: postgresql-10.23-3.module+el8.9.0+20831+78489779.x86_64 2024-01-24T14:17:56.277 INFO:teuthology.orchestra.run.smithi194.stdout: postgresql-contrib-10.23-3.module+el8.9.0+20831+78489779.x86_64 2024-01-24T14:17:56.277 INFO:teuthology.orchestra.run.smithi194.stdout: postgresql-server-10.23-3.module+el8.9.0+20831+78489779.x86_64 2024-01-24T14:17:56.278 INFO:teuthology.orchestra.run.smithi194.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-01-24T14:17:56.278 INFO:teuthology.orchestra.run.smithi194.stdout: python3-babel-2.5.1-7.el8.noarch 2024-01-24T14:17:56.278 INFO:teuthology.orchestra.run.smithi194.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-01-24T14:17:56.278 INFO:teuthology.orchestra.run.smithi194.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-01-24T14:17:56.278 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-01-24T14:17:56.278 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-argparse-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:56.278 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-common-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:56.278 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cephfs-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:56.278 INFO:teuthology.orchestra.run.smithi194.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-01-24T14:17:56.278 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-01-24T14:17:56.278 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-01-24T14:17:56.278 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-01-24T14:17:56.278 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2024-01-24T14:17:56.278 INFO:teuthology.orchestra.run.smithi194.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-01-24T14:17:56.278 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-6.2-6.el8.noarch 2024-01-24T14:17:56.279 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-01-24T14:17:56.279 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-01-24T14:17:56.279 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-01-24T14:17:56.279 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-01-24T14:17:56.279 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-01-24T14:17:56.279 INFO:teuthology.orchestra.run.smithi194.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-01-24T14:17:56.279 INFO:teuthology.orchestra.run.smithi194.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-01-24T14:17:56.279 INFO:teuthology.orchestra.run.smithi194.stdout: python3-mako-1.0.6-14.el8.noarch 2024-01-24T14:17:56.279 INFO:teuthology.orchestra.run.smithi194.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-01-24T14:17:56.279 INFO:teuthology.orchestra.run.smithi194.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-01-24T14:17:56.279 INFO:teuthology.orchestra.run.smithi194.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-01-24T14:17:56.279 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-01-24T14:17:56.279 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-01-24T14:17:56.279 INFO:teuthology.orchestra.run.smithi194.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-01-24T14:17:56.280 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-01-24T14:17:56.280 INFO:teuthology.orchestra.run.smithi194.stdout: python3-portend-2.6-1.el8.noarch 2024-01-24T14:17:56.280 INFO:teuthology.orchestra.run.smithi194.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-01-24T14:17:56.280 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-01-24T14:17:56.280 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-01-24T14:17:56.280 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-01-24T14:17:56.280 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pycparser-2.14-14.el8.noarch 2024-01-24T14:17:56.280 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pytz-2017.2-11.el8.noarch 2024-01-24T14:17:56.280 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rados-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:56.280 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rbd-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:56.280 INFO:teuthology.orchestra.run.smithi194.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-01-24T14:17:56.280 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-01-24T14:17:56.280 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rgw-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:56.280 INFO:teuthology.orchestra.run.smithi194.stdout: python3-routes-2.4.1-12.el8.noarch 2024-01-24T14:17:56.281 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rsa-4.9-2.el8.noarch 2024-01-24T14:17:56.281 INFO:teuthology.orchestra.run.smithi194.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-01-24T14:17:56.281 INFO:teuthology.orchestra.run.smithi194.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-01-24T14:17:56.281 INFO:teuthology.orchestra.run.smithi194.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-01-24T14:17:56.281 INFO:teuthology.orchestra.run.smithi194.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-01-24T14:17:56.281 INFO:teuthology.orchestra.run.smithi194.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-01-24T14:17:56.281 INFO:teuthology.orchestra.run.smithi194.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-01-24T14:17:56.281 INFO:teuthology.orchestra.run.smithi194.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-01-24T14:17:56.281 INFO:teuthology.orchestra.run.smithi194.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-01-24T14:17:56.281 INFO:teuthology.orchestra.run.smithi194.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-01-24T14:17:56.281 INFO:teuthology.orchestra.run.smithi194.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-01-24T14:17:56.281 INFO:teuthology.orchestra.run.smithi194.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-01-24T14:17:56.281 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-fuse-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:56.281 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:56.281 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-nbd-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:17:56.282 INFO:teuthology.orchestra.run.smithi194.stdout: socat-1.7.4.1-1.el8.x86_64 2024-01-24T14:17:56.282 INFO:teuthology.orchestra.run.smithi194.stdout: thrift-0.13.0-2.el8.x86_64 2024-01-24T14:17:56.282 INFO:teuthology.orchestra.run.smithi194.stdout: uuid-1.6.2-43.el8.x86_64 2024-01-24T14:17:56.282 INFO:teuthology.orchestra.run.smithi194.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2024-01-24T14:17:56.282 INFO:teuthology.orchestra.run.smithi194.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-01-24T14:17:56.282 INFO:teuthology.orchestra.run.smithi194.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2024-01-24T14:17:56.282 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-24T14:17:56.282 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-24T14:17:57.452 DEBUG:teuthology.parallel:result is None 2024-01-24T14:18:04.809 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 161/161 2024-01-24T14:18:04.810 INFO:teuthology.orchestra.run.smithi019.stdout:Installed products updated. 2024-01-24T14:18:05.497 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:18:05.497 INFO:teuthology.orchestra.run.smithi019.stdout:Upgraded: 2024-01-24T14:18:05.498 INFO:teuthology.orchestra.run.smithi019.stdout: elfutils-libelf-0.189-3.el8.x86_64 2024-01-24T14:18:05.498 INFO:teuthology.orchestra.run.smithi019.stdout: elfutils-libs-0.189-3.el8.x86_64 2024-01-24T14:18:05.498 INFO:teuthology.orchestra.run.smithi019.stdout: iproute-6.2.0-5.el8_9.x86_64 2024-01-24T14:18:05.498 INFO:teuthology.orchestra.run.smithi019.stdout: libblkid-2.32.1-43.el8.x86_64 2024-01-24T14:18:05.498 INFO:teuthology.orchestra.run.smithi019.stdout: libfdisk-2.32.1-43.el8.x86_64 2024-01-24T14:18:05.498 INFO:teuthology.orchestra.run.smithi019.stdout: libmount-2.32.1-43.el8.x86_64 2024-01-24T14:18:05.498 INFO:teuthology.orchestra.run.smithi019.stdout: librados2-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:18:05.498 INFO:teuthology.orchestra.run.smithi019.stdout: librbd1-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:18:05.498 INFO:teuthology.orchestra.run.smithi019.stdout: libsmartcols-2.32.1-43.el8.x86_64 2024-01-24T14:18:05.498 INFO:teuthology.orchestra.run.smithi019.stdout: libuuid-2.32.1-43.el8.x86_64 2024-01-24T14:18:05.498 INFO:teuthology.orchestra.run.smithi019.stdout: openssl-1:1.1.1k-12.el8_9.x86_64 2024-01-24T14:18:05.498 INFO:teuthology.orchestra.run.smithi019.stdout: openssl-libs-1:1.1.1k-12.el8_9.x86_64 2024-01-24T14:18:05.498 INFO:teuthology.orchestra.run.smithi019.stdout: util-linux-2.32.1-43.el8.x86_64 2024-01-24T14:18:05.499 INFO:teuthology.orchestra.run.smithi019.stdout: util-linux-user-2.32.1-43.el8.x86_64 2024-01-24T14:18:05.499 INFO:teuthology.orchestra.run.smithi019.stdout: xfsdump-3.1.8-7.el8_9.x86_64 2024-01-24T14:18:05.499 INFO:teuthology.orchestra.run.smithi019.stdout: xfsprogs-5.0.0-12.el8.x86_64 2024-01-24T14:18:05.499 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-01-24T14:18:05.499 INFO:teuthology.orchestra.run.smithi019.stdout: NetworkManager-initscripts-updown-1:1.40.16-13.el8_9.noarch 2024-01-24T14:18:05.499 INFO:teuthology.orchestra.run.smithi019.stdout: automake-1.16.1-8.el8.noarch 2024-01-24T14:18:05.499 INFO:teuthology.orchestra.run.smithi019.stdout: bison-3.0.4-10.el8.x86_64 2024-01-24T14:18:05.499 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:18:05.499 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-base-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:18:05.499 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-common-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:18:05.499 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-fuse-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:18:05.499 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-grafana-dashboards-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T14:18:05.499 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-immutable-object-cache-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:18:05.500 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mds-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:18:05.500 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:18:05.500 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-cephadm-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T14:18:05.500 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-dashboard-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T14:18:05.500 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-diskprediction-local-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T14:18:05.500 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-modules-core-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T14:18:05.500 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-rook-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T14:18:05.500 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mon-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:18:05.500 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-osd-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:18:05.500 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-prometheus-alerts-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T14:18:05.500 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-radosgw-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:18:05.501 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-selinux-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:18:05.501 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-test-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:18:05.501 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-volume-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T14:18:05.501 INFO:teuthology.orchestra.run.smithi019.stdout: cephadm-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T14:18:05.501 INFO:teuthology.orchestra.run.smithi019.stdout: cephfs-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:18:05.501 INFO:teuthology.orchestra.run.smithi019.stdout: cephfs-top-2:18.2.1-341.g4e3182a6.el8.noarch 2024-01-24T14:18:05.501 INFO:teuthology.orchestra.run.smithi019.stdout: elfutils-libelf-devel-0.189-3.el8.x86_64 2024-01-24T14:18:05.501 INFO:teuthology.orchestra.run.smithi019.stdout: flex-2.6.1-9.el8.x86_64 2024-01-24T14:18:05.501 INFO:teuthology.orchestra.run.smithi019.stdout: fuse-2.9.7-17.el8.x86_64 2024-01-24T14:18:05.501 INFO:teuthology.orchestra.run.smithi019.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-01-24T14:18:05.501 INFO:teuthology.orchestra.run.smithi019.stdout: jq-1.6-7.el8.x86_64 2024-01-24T14:18:05.502 INFO:teuthology.orchestra.run.smithi019.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2024-01-24T14:18:05.502 INFO:teuthology.orchestra.run.smithi019.stdout: krb5-devel-1.18.2-26.el8_9.x86_64 2024-01-24T14:18:05.502 INFO:teuthology.orchestra.run.smithi019.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-01-24T14:18:05.502 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs-devel-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:18:05.502 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs2-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:18:05.502 INFO:teuthology.orchestra.run.smithi019.stdout: libcephsqlite-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:18:05.502 INFO:teuthology.orchestra.run.smithi019.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2024-01-24T14:18:05.502 INFO:teuthology.orchestra.run.smithi019.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-01-24T14:18:05.502 INFO:teuthology.orchestra.run.smithi019.stdout: liboath-2.6.2-3.el8.x86_64 2024-01-24T14:18:05.502 INFO:teuthology.orchestra.run.smithi019.stdout: libpq-13.11-1.el8.x86_64 2024-01-24T14:18:05.502 INFO:teuthology.orchestra.run.smithi019.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-01-24T14:18:05.502 INFO:teuthology.orchestra.run.smithi019.stdout: librabbitmq-0.9.0-4.el8.x86_64 2024-01-24T14:18:05.502 INFO:teuthology.orchestra.run.smithi019.stdout: librados-devel-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:18:05.503 INFO:teuthology.orchestra.run.smithi019.stdout: libradosstriper1-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:18:05.503 INFO:teuthology.orchestra.run.smithi019.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-01-24T14:18:05.503 INFO:teuthology.orchestra.run.smithi019.stdout: librgw2-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:18:05.503 INFO:teuthology.orchestra.run.smithi019.stdout: libselinux-devel-2.9-5.el8.x86_64 2024-01-24T14:18:05.503 INFO:teuthology.orchestra.run.smithi019.stdout: libsepol-devel-2.9-3.el8.x86_64 2024-01-24T14:18:05.503 INFO:teuthology.orchestra.run.smithi019.stdout: libtool-2.4.6-25.el8.x86_64 2024-01-24T14:18:05.503 INFO:teuthology.orchestra.run.smithi019.stdout: libunwind-1.3.1-3.el8.x86_64 2024-01-24T14:18:05.503 INFO:teuthology.orchestra.run.smithi019.stdout: libuuid-devel-2.32.1-43.el8.x86_64 2024-01-24T14:18:05.503 INFO:teuthology.orchestra.run.smithi019.stdout: libverto-devel-0.3.0-5.el8.x86_64 2024-01-24T14:18:05.503 INFO:teuthology.orchestra.run.smithi019.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2024-01-24T14:18:05.504 INFO:teuthology.orchestra.run.smithi019.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-01-24T14:18:05.504 INFO:teuthology.orchestra.run.smithi019.stdout: oniguruma-6.8.2-2.el8.x86_64 2024-01-24T14:18:05.504 INFO:teuthology.orchestra.run.smithi019.stdout: openblas-0.3.15-6.el8.x86_64 2024-01-24T14:18:05.504 INFO:teuthology.orchestra.run.smithi019.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-01-24T14:18:05.504 INFO:teuthology.orchestra.run.smithi019.stdout: openssl-devel-1:1.1.1k-12.el8_9.x86_64 2024-01-24T14:18:05.504 INFO:teuthology.orchestra.run.smithi019.stdout: pcre2-devel-10.32-2.el8.x86_64 2024-01-24T14:18:05.504 INFO:teuthology.orchestra.run.smithi019.stdout: pcre2-utf16-10.32-2.el8.x86_64 2024-01-24T14:18:05.504 INFO:teuthology.orchestra.run.smithi019.stdout: pcre2-utf32-10.32-2.el8.x86_64 2024-01-24T14:18:05.505 INFO:teuthology.orchestra.run.smithi019.stdout: postgresql-10.23-3.module+el8.9.0+20831+78489779.x86_64 2024-01-24T14:18:05.505 INFO:teuthology.orchestra.run.smithi019.stdout: postgresql-contrib-10.23-3.module+el8.9.0+20831+78489779.x86_64 2024-01-24T14:18:05.505 INFO:teuthology.orchestra.run.smithi019.stdout: postgresql-server-10.23-3.module+el8.9.0+20831+78489779.x86_64 2024-01-24T14:18:05.505 INFO:teuthology.orchestra.run.smithi019.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-01-24T14:18:05.505 INFO:teuthology.orchestra.run.smithi019.stdout: python3-babel-2.5.1-7.el8.noarch 2024-01-24T14:18:05.505 INFO:teuthology.orchestra.run.smithi019.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-01-24T14:18:05.505 INFO:teuthology.orchestra.run.smithi019.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-01-24T14:18:05.505 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-01-24T14:18:05.505 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-argparse-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:18:05.506 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-common-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:18:05.506 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cephfs-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:18:05.506 INFO:teuthology.orchestra.run.smithi019.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-01-24T14:18:05.506 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-01-24T14:18:05.506 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-01-24T14:18:05.506 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-01-24T14:18:05.506 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2024-01-24T14:18:05.506 INFO:teuthology.orchestra.run.smithi019.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-01-24T14:18:05.506 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-6.2-6.el8.noarch 2024-01-24T14:18:05.507 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-01-24T14:18:05.507 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-01-24T14:18:05.507 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-01-24T14:18:05.507 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-01-24T14:18:05.507 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-01-24T14:18:05.507 INFO:teuthology.orchestra.run.smithi019.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-01-24T14:18:05.507 INFO:teuthology.orchestra.run.smithi019.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-01-24T14:18:05.507 INFO:teuthology.orchestra.run.smithi019.stdout: python3-mako-1.0.6-14.el8.noarch 2024-01-24T14:18:05.507 INFO:teuthology.orchestra.run.smithi019.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-01-24T14:18:05.507 INFO:teuthology.orchestra.run.smithi019.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-01-24T14:18:05.507 INFO:teuthology.orchestra.run.smithi019.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-01-24T14:18:05.507 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-01-24T14:18:05.508 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-01-24T14:18:05.508 INFO:teuthology.orchestra.run.smithi019.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-01-24T14:18:05.508 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-01-24T14:18:05.508 INFO:teuthology.orchestra.run.smithi019.stdout: python3-portend-2.6-1.el8.noarch 2024-01-24T14:18:05.508 INFO:teuthology.orchestra.run.smithi019.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-01-24T14:18:05.509 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-01-24T14:18:05.509 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-01-24T14:18:05.509 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-01-24T14:18:05.509 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pycparser-2.14-14.el8.noarch 2024-01-24T14:18:05.509 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pytz-2017.2-11.el8.noarch 2024-01-24T14:18:05.509 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rados-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:18:05.510 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rbd-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:18:05.510 INFO:teuthology.orchestra.run.smithi019.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-01-24T14:18:05.510 INFO:teuthology.orchestra.run.smithi019.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-01-24T14:18:05.510 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rgw-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:18:05.510 INFO:teuthology.orchestra.run.smithi019.stdout: python3-routes-2.4.1-12.el8.noarch 2024-01-24T14:18:05.510 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rsa-4.9-2.el8.noarch 2024-01-24T14:18:05.511 INFO:teuthology.orchestra.run.smithi019.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-01-24T14:18:05.511 INFO:teuthology.orchestra.run.smithi019.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-01-24T14:18:05.511 INFO:teuthology.orchestra.run.smithi019.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-01-24T14:18:05.511 INFO:teuthology.orchestra.run.smithi019.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-01-24T14:18:05.511 INFO:teuthology.orchestra.run.smithi019.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-01-24T14:18:05.511 INFO:teuthology.orchestra.run.smithi019.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-01-24T14:18:05.511 INFO:teuthology.orchestra.run.smithi019.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-01-24T14:18:05.511 INFO:teuthology.orchestra.run.smithi019.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-01-24T14:18:05.511 INFO:teuthology.orchestra.run.smithi019.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-01-24T14:18:05.511 INFO:teuthology.orchestra.run.smithi019.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-01-24T14:18:05.511 INFO:teuthology.orchestra.run.smithi019.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-01-24T14:18:05.511 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-fuse-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:18:05.512 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-mirror-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:18:05.512 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-nbd-2:18.2.1-341.g4e3182a6.el8.x86_64 2024-01-24T14:18:05.512 INFO:teuthology.orchestra.run.smithi019.stdout: socat-1.7.4.1-1.el8.x86_64 2024-01-24T14:18:05.512 INFO:teuthology.orchestra.run.smithi019.stdout: thrift-0.13.0-2.el8.x86_64 2024-01-24T14:18:05.512 INFO:teuthology.orchestra.run.smithi019.stdout: uuid-1.6.2-43.el8.x86_64 2024-01-24T14:18:05.516 INFO:teuthology.orchestra.run.smithi019.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2024-01-24T14:18:05.516 INFO:teuthology.orchestra.run.smithi019.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-01-24T14:18:05.516 INFO:teuthology.orchestra.run.smithi019.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2024-01-24T14:18:05.516 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:18:05.517 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-01-24T14:18:05.733 DEBUG:teuthology.parallel:result is None 2024-01-24T14:18:05.733 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4e3182a65800aed6d4aa2f6b86da3cfd72188134 2024-01-24T14:18:05.877 DEBUG:teuthology.orchestra.run.smithi019:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-24T14:18:05.905 INFO:teuthology.orchestra.run.smithi019.stdout:18.2.1-341.g4e3182a6.el8 2024-01-24T14:18:05.906 INFO:teuthology.packaging:The installed version of ceph is 18.2.1-341.g4e3182a6.el8 2024-01-24T14:18:05.906 INFO:teuthology.task.install:The correct ceph version 18.2.1-341.g4e3182a6 is installed. 2024-01-24T14:18:05.908 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4e3182a65800aed6d4aa2f6b86da3cfd72188134 2024-01-24T14:18:06.051 DEBUG:teuthology.orchestra.run.smithi123:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-24T14:18:06.080 INFO:teuthology.orchestra.run.smithi123.stdout:18.2.1-341.g4e3182a6.el8 2024-01-24T14:18:06.080 INFO:teuthology.packaging:The installed version of ceph is 18.2.1-341.g4e3182a6.el8 2024-01-24T14:18:06.081 INFO:teuthology.task.install:The correct ceph version 18.2.1-341.g4e3182a6 is installed. 2024-01-24T14:18:06.082 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4e3182a65800aed6d4aa2f6b86da3cfd72188134 2024-01-24T14:18:06.377 DEBUG:teuthology.orchestra.run.smithi194:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-24T14:18:06.405 INFO:teuthology.orchestra.run.smithi194.stdout:18.2.1-341.g4e3182a6.el8 2024-01-24T14:18:06.406 INFO:teuthology.packaging:The installed version of ceph is 18.2.1-341.g4e3182a6.el8 2024-01-24T14:18:06.406 INFO:teuthology.task.install:The correct ceph version 18.2.1-341.g4e3182a6 is installed. 2024-01-24T14:18:06.408 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-01-24T14:18:06.408 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-01-24T14:18:06.408 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-24T14:18:06.461 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-01-24T14:18:06.461 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-24T14:18:06.510 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-01-24T14:18:06.510 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-24T14:18:06.559 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-01-24T14:18:06.559 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-01-24T14:18:06.559 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/usr/bin/daemon-helper 2024-01-24T14:18:06.594 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-24T14:18:06.669 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-01-24T14:18:06.669 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/daemon-helper 2024-01-24T14:18:06.703 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-24T14:18:06.775 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-01-24T14:18:06.775 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/daemon-helper 2024-01-24T14:18:06.806 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-24T14:18:06.880 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-01-24T14:18:06.881 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-01-24T14:18:06.881 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-24T14:18:06.915 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-24T14:18:06.990 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-01-24T14:18:06.990 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-24T14:18:07.022 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-24T14:18:07.092 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-01-24T14:18:07.092 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-24T14:18:07.123 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-24T14:18:07.193 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-01-24T14:18:07.193 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-01-24T14:18:07.193 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/usr/bin/stdin-killer 2024-01-24T14:18:07.227 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-24T14:18:07.301 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-01-24T14:18:07.302 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/stdin-killer 2024-01-24T14:18:07.334 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-24T14:18:07.407 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-01-24T14:18:07.407 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/stdin-killer 2024-01-24T14:18:07.438 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-24T14:18:07.508 INFO:teuthology.run_tasks:Running task cephadm... 2024-01-24T14:18:07.586 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '4e3182a65800aed6d4aa2f6b86da3cfd72188134', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-01-24T14:18:07.586 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 2024-01-24T14:18:07.587 INFO:tasks.cephadm:Cluster fsid is 654b0c8e-bac3-11ee-95b1-87774f69a715 2024-01-24T14:18:07.587 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-01-24T14:18:07.587 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.19', 'mon.b': '172.21.15.123', 'mon.c': '172.21.15.194'} 2024-01-24T14:18:07.587 INFO:tasks.cephadm:First mon is mon.a on smithi019 2024-01-24T14:18:07.587 INFO:tasks.cephadm:First mgr is x 2024-01-24T14:18:07.587 INFO:tasks.cephadm:Normalizing hostnames... 2024-01-24T14:18:07.587 DEBUG:teuthology.orchestra.run.smithi019:> sudo hostname $(hostname -s) 2024-01-24T14:18:07.623 DEBUG:teuthology.orchestra.run.smithi123:> sudo hostname $(hostname -s) 2024-01-24T14:18:07.657 DEBUG:teuthology.orchestra.run.smithi194:> sudo hostname $(hostname -s) 2024-01-24T14:18:07.688 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-01-24T14:18:07.688 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4e3182a65800aed6d4aa2f6b86da3cfd72188134 2024-01-24T14:18:07.954 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '4e3182a65800aed6d4aa2f6b86da3cfd72188134', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/76187/', 'root_build_cause': 'SCMTRIGGER', 'version': '18.2.1-341-g4e3182a6', 'node_name': '172.21.2.7+braggi07', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '18.2.1-341.g4e3182a6'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-vshankar-testing1-reef-2024-01-24-1150/4e3182a65800aed6d4aa2f6b86da3cfd72188134/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2024-01-24 07:52:51.100568', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-vshankar-testing1-reef-2024-01-24-1150', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-vshankar-testing1-reef-2024-01-24-1150/4e3182a65800aed6d4aa2f6b86da3cfd72188134/centos/8/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-01-24T14:18:08.205 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-vshankar-testing1-reef-2024-01-24-1150, sha1 4e3182a65800aed6d4aa2f6b86da3cfd72188134 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F8%2Fx86_64&flavor=default&sha1=4e3182a65800aed6d4aa2f6b86da3cfd72188134 2024-01-24T14:18:08.206 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-vshankar-testing1-reef-2024-01-24-1150/4e3182a65800aed6d4aa2f6b86da3cfd72188134/centos/8/x86_64/flavors/default/cephadm 2024-01-24T14:18:08.206 DEBUG:teuthology.orchestra.run.smithi019:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-vshankar-testing1-reef-2024-01-24-1150/4e3182a65800aed6d4aa2f6b86da3cfd72188134/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-24T14:18:08.641 INFO:teuthology.orchestra.run.smithi019.stdout:-rw-rw-r--. 1 ubuntu ubuntu 690687 Jan 24 14:18 /home/ubuntu/cephtest/cephadm 2024-01-24T14:18:08.641 DEBUG:teuthology.orchestra.run.smithi123:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-vshankar-testing1-reef-2024-01-24-1150/4e3182a65800aed6d4aa2f6b86da3cfd72188134/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-24T14:18:09.032 INFO:teuthology.orchestra.run.smithi123.stdout:-rw-rw-r--. 1 ubuntu ubuntu 690687 Jan 24 14:18 /home/ubuntu/cephtest/cephadm 2024-01-24T14:18:09.033 DEBUG:teuthology.orchestra.run.smithi194:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-vshankar-testing1-reef-2024-01-24-1150/4e3182a65800aed6d4aa2f6b86da3cfd72188134/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-24T14:18:10.117 INFO:teuthology.orchestra.run.smithi194.stdout:-rw-rw-r--. 1 ubuntu ubuntu 690687 Jan 24 14:18 /home/ubuntu/cephtest/cephadm 2024-01-24T14:18:10.117 DEBUG:teuthology.orchestra.run.smithi019:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-24T14:18:10.140 DEBUG:teuthology.orchestra.run.smithi123:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-24T14:18:10.163 DEBUG:teuthology.orchestra.run.smithi194:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-24T14:18:10.201 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 on all hosts... 2024-01-24T14:18:10.201 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 pull 2024-01-24T14:18:10.203 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 pull 2024-01-24T14:18:10.205 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 pull 2024-01-24T14:18:11.957 INFO:teuthology.orchestra.run.smithi194.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134... 2024-01-24T14:18:11.974 INFO:teuthology.orchestra.run.smithi123.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134... 2024-01-24T14:18:13.748 INFO:teuthology.orchestra.run.smithi019.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134... 2024-01-24T14:18:51.001 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-01-24T14:18:51.001 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph_version": "ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)", 2024-01-24T14:18:51.001 INFO:teuthology.orchestra.run.smithi019.stdout: "image_id": "d650b791ad541e9330061e00985d36774db72ca0aaab13a27fd81b9f0f9eec74", 2024-01-24T14:18:51.001 INFO:teuthology.orchestra.run.smithi019.stdout: "repo_digests": [ 2024-01-24T14:18:51.001 INFO:teuthology.orchestra.run.smithi019.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:22efea2f35520d1212d406fc0d380a7fe7894a099a64f04365d740d4e7cb8285" 2024-01-24T14:18:51.002 INFO:teuthology.orchestra.run.smithi019.stdout: ] 2024-01-24T14:18:51.002 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-01-24T14:18:59.770 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-01-24T14:18:59.771 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_version": "ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)", 2024-01-24T14:18:59.771 INFO:teuthology.orchestra.run.smithi123.stdout: "image_id": "d650b791ad541e9330061e00985d36774db72ca0aaab13a27fd81b9f0f9eec74", 2024-01-24T14:18:59.771 INFO:teuthology.orchestra.run.smithi123.stdout: "repo_digests": [ 2024-01-24T14:18:59.771 INFO:teuthology.orchestra.run.smithi123.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:22efea2f35520d1212d406fc0d380a7fe7894a099a64f04365d740d4e7cb8285" 2024-01-24T14:18:59.771 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-01-24T14:18:59.771 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-01-24T14:19:04.014 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-01-24T14:19:04.014 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_version": "ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)", 2024-01-24T14:19:04.015 INFO:teuthology.orchestra.run.smithi194.stdout: "image_id": "d650b791ad541e9330061e00985d36774db72ca0aaab13a27fd81b9f0f9eec74", 2024-01-24T14:19:04.015 INFO:teuthology.orchestra.run.smithi194.stdout: "repo_digests": [ 2024-01-24T14:19:04.015 INFO:teuthology.orchestra.run.smithi194.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:22efea2f35520d1212d406fc0d380a7fe7894a099a64f04365d740d4e7cb8285" 2024-01-24T14:19:04.015 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-01-24T14:19:04.015 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-01-24T14:19:04.034 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /etc/ceph 2024-01-24T14:19:04.076 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /etc/ceph 2024-01-24T14:19:04.114 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /etc/ceph 2024-01-24T14:19:04.144 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 777 /etc/ceph 2024-01-24T14:19:04.179 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 777 /etc/ceph 2024-01-24T14:19:04.212 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 777 /etc/ceph 2024-01-24T14:19:04.239 INFO:tasks.cephadm:Writing seed config... 2024-01-24T14:19:04.240 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-01-24T14:19:04.240 INFO:tasks.cephadm: override: [client] debug client = 20 2024-01-24T14:19:04.240 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-01-24T14:19:04.240 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-01-24T14:19:04.240 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-01-24T14:19:04.240 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-01-24T14:19:04.240 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-01-24T14:19:04.240 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-01-24T14:19:04.241 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-01-24T14:19:04.241 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-01-24T14:19:04.241 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-01-24T14:19:04.241 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-01-24T14:19:04.241 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-01-24T14:19:04.241 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-01-24T14:19:04.241 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-01-24T14:19:04.241 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-01-24T14:19:04.241 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-01-24T14:19:04.241 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-01-24T14:19:04.241 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-01-24T14:19:04.241 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-01-24T14:19:04.241 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-01-24T14:19:04.241 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-01-24T14:19:04.241 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-01-24T14:19:04.242 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-01-24T14:19:04.242 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-01-24T14:19:04.242 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-01-24T14:19:04.242 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-01-24T14:19:04.242 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-01-24T14:19:04.242 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-01-24T14:19:04.242 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-01-24T14:19:04.242 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-01-24T14:19:04.242 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-01-24T14:19:04.242 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-01-24T14:19:04.242 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-01-24T14:19:04.242 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-01-24T14:19:04.242 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-01-24T14:19:04.242 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-01-24T14:19:04.242 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-01-24T14:19:04.243 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-01-24T14:19:04.243 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-01-24T14:19:04.243 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-01-24T14:19:04.243 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-01-24T14:19:04.243 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-01-24T14:19:04.243 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-01-24T14:19:04.243 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-01-24T14:19:04.244 DEBUG:teuthology.orchestra.run.smithi019:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-01-24T14:19:04.266 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 654b0c8e-bac3-11ee-95b1-87774f69a715 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-01-24T14:19:04.266 DEBUG:teuthology.orchestra.run.smithi019:mon.a> sudo journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mon.a.service 2024-01-24T14:19:04.308 DEBUG:teuthology.orchestra.run.smithi019:mgr.x> sudo journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mgr.x.service 2024-01-24T14:19:04.343 INFO:journalctl@ceph.mon.a.smithi019.stdout:-- Logs begin at Wed 2024-01-24 14:03:52 UTC. -- 2024-01-24T14:19:04.344 INFO:tasks.cephadm:Bootstrapping... 2024-01-24T14:19:04.344 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 -v bootstrap --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.19 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-01-24T14:19:04.376 INFO:journalctl@ceph.mgr.x.smithi019.stdout:-- Logs begin at Wed 2024-01-24 14:03:52 UTC. -- 2024-01-24T14:19:04.533 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-01-24T14:19:04.533 INFO:teuthology.orchestra.run.smithi019.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134', '-v', 'bootstrap', '--fsid', '654b0c8e-bac3-11ee-95b1-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.19', '--skip-admin-label'] 2024-01-24T14:19:04.565 INFO:teuthology.orchestra.run.smithi019.stdout:/bin/podman: stdout 4.6.1 2024-01-24T14:19:04.565 INFO:teuthology.orchestra.run.smithi019.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-01-24T14:19:04.565 INFO:teuthology.orchestra.run.smithi019.stdout:Verifying podman|docker is present... 2024-01-24T14:19:04.603 INFO:teuthology.orchestra.run.smithi019.stdout:/bin/podman: stdout 4.6.1 2024-01-24T14:19:04.603 INFO:teuthology.orchestra.run.smithi019.stdout:Verifying lvm2 is present... 2024-01-24T14:19:04.603 INFO:teuthology.orchestra.run.smithi019.stdout:Verifying time synchronization is in place... 2024-01-24T14:19:04.611 INFO:teuthology.orchestra.run.smithi019.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-01-24T14:19:04.611 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-01-24T14:19:04.617 INFO:teuthology.orchestra.run.smithi019.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-01-24T14:19:04.617 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stdout inactive 2024-01-24T14:19:04.624 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stdout enabled 2024-01-24T14:19:04.630 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stdout active 2024-01-24T14:19:04.631 INFO:teuthology.orchestra.run.smithi019.stdout:Unit chronyd.service is enabled and running 2024-01-24T14:19:04.631 INFO:teuthology.orchestra.run.smithi019.stdout:Repeating the final host check... 2024-01-24T14:19:04.667 INFO:teuthology.orchestra.run.smithi019.stdout:/bin/podman: stdout 4.6.1 2024-01-24T14:19:04.667 INFO:teuthology.orchestra.run.smithi019.stdout:podman (/bin/podman) version 4.6.1 is present 2024-01-24T14:19:04.667 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl is present 2024-01-24T14:19:04.667 INFO:teuthology.orchestra.run.smithi019.stdout:lvcreate is present 2024-01-24T14:19:04.674 INFO:teuthology.orchestra.run.smithi019.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-01-24T14:19:04.674 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-01-24T14:19:04.680 INFO:teuthology.orchestra.run.smithi019.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-01-24T14:19:04.680 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stdout inactive 2024-01-24T14:19:04.687 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stdout enabled 2024-01-24T14:19:04.694 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stdout active 2024-01-24T14:19:04.694 INFO:teuthology.orchestra.run.smithi019.stdout:Unit chronyd.service is enabled and running 2024-01-24T14:19:04.694 INFO:teuthology.orchestra.run.smithi019.stdout:Host looks OK 2024-01-24T14:19:04.694 INFO:teuthology.orchestra.run.smithi019.stdout:Cluster fsid: 654b0c8e-bac3-11ee-95b1-87774f69a715 2024-01-24T14:19:04.694 INFO:teuthology.orchestra.run.smithi019.stdout:Acquiring lock 139653425981928 on /run/cephadm/654b0c8e-bac3-11ee-95b1-87774f69a715.lock 2024-01-24T14:19:04.694 INFO:teuthology.orchestra.run.smithi019.stdout:Lock 139653425981928 acquired on /run/cephadm/654b0c8e-bac3-11ee-95b1-87774f69a715.lock 2024-01-24T14:19:04.694 INFO:teuthology.orchestra.run.smithi019.stdout:Verifying IP 172.21.15.19 port 3300 ... 2024-01-24T14:19:04.695 INFO:teuthology.orchestra.run.smithi019.stdout:Verifying IP 172.21.15.19 port 6789 ... 2024-01-24T14:19:04.695 INFO:teuthology.orchestra.run.smithi019.stdout:Base mon IP(s) is [172.21.15.19:3300, 172.21.15.19:6789], mon addrv is [v2:172.21.15.19:3300,v1:172.21.15.19:6789] 2024-01-24T14:19:04.699 INFO:teuthology.orchestra.run.smithi019.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp metric 100 2024-01-24T14:19:04.699 INFO:teuthology.orchestra.run.smithi019.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.19 metric 100 2024-01-24T14:19:04.703 INFO:teuthology.orchestra.run.smithi019.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-01-24T14:19:04.703 INFO:teuthology.orchestra.run.smithi019.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2024-01-24T14:19:04.703 INFO:teuthology.orchestra.run.smithi019.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1768sec hoplimit 64 pref medium 2024-01-24T14:19:04.706 INFO:teuthology.orchestra.run.smithi019.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-01-24T14:19:04.706 INFO:teuthology.orchestra.run.smithi019.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-01-24T14:19:04.706 INFO:teuthology.orchestra.run.smithi019.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-01-24T14:19:04.706 INFO:teuthology.orchestra.run.smithi019.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-01-24T14:19:04.706 INFO:teuthology.orchestra.run.smithi019.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:161e/64 scope link 2024-01-24T14:19:04.706 INFO:teuthology.orchestra.run.smithi019.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-01-24T14:19:04.707 INFO:teuthology.orchestra.run.smithi019.stdout:Mon IP `172.21.15.19` is in CIDR network `172.21.0.0/20` 2024-01-24T14:19:04.707 INFO:teuthology.orchestra.run.smithi019.stdout:Mon IP `172.21.15.19` is in CIDR network `172.21.0.0/20` 2024-01-24T14:19:04.707 INFO:teuthology.orchestra.run.smithi019.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-01-24T14:19:04.708 INFO:teuthology.orchestra.run.smithi019.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-01-24T14:19:04.709 INFO:teuthology.orchestra.run.smithi019.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134... 2024-01-24T14:19:05.100 INFO:teuthology.orchestra.run.smithi019.stdout:/bin/podman: stdout d650b791ad541e9330061e00985d36774db72ca0aaab13a27fd81b9f0f9eec74 2024-01-24T14:19:05.100 INFO:teuthology.orchestra.run.smithi019.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134... 2024-01-24T14:19:05.100 INFO:teuthology.orchestra.run.smithi019.stdout:/bin/podman: stderr Getting image source signatures 2024-01-24T14:19:05.100 INFO:teuthology.orchestra.run.smithi019.stdout:/bin/podman: stderr Copying blob sha256:5bd92459d7edc3325b5efa151665fd479e83585b213707287afb2af065f59941 2024-01-24T14:19:05.100 INFO:teuthology.orchestra.run.smithi019.stdout:/bin/podman: stderr Copying blob sha256:ea3366b8a90265ae3f6d3736cb0de4b52da4ffec919b46c6cb5bc5be1372efe8 2024-01-24T14:19:05.100 INFO:teuthology.orchestra.run.smithi019.stdout:/bin/podman: stderr Copying config sha256:d650b791ad541e9330061e00985d36774db72ca0aaab13a27fd81b9f0f9eec74 2024-01-24T14:19:05.100 INFO:teuthology.orchestra.run.smithi019.stdout:/bin/podman: stderr Writing manifest to image destination 2024-01-24T14:19:05.858 INFO:teuthology.orchestra.run.smithi019.stdout:ceph: stdout ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable) 2024-01-24T14:19:05.858 INFO:teuthology.orchestra.run.smithi019.stdout:Ceph version: ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable) 2024-01-24T14:19:05.858 INFO:teuthology.orchestra.run.smithi019.stdout:Extracting ceph user uid/gid from container image... 2024-01-24T14:19:06.440 INFO:teuthology.orchestra.run.smithi019.stdout:stat: stdout 167 167 2024-01-24T14:19:06.440 INFO:teuthology.orchestra.run.smithi019.stdout:Creating initial keys... 2024-01-24T14:19:07.073 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph-authtool: stdout AQBaHLFlC5qOKRAAKF69EBlKGfoMiGtS4O8Oow== 2024-01-24T14:19:07.674 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph-authtool: stdout AQBbHLFlfe2sFBAA7TIRyBrlnGnCk/nIsUJ3rQ== 2024-01-24T14:19:08.296 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph-authtool: stdout AQBbHLFlFGHdOBAADWQRrJCSDsNcjf6fRay4PQ== 2024-01-24T14:19:08.297 INFO:teuthology.orchestra.run.smithi019.stdout:Creating initial monmap... 2024-01-24T14:19:08.899 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-01-24T14:19:08.899 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2024-01-24T14:19:08.899 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 654b0c8e-bac3-11ee-95b1-87774f69a715 2024-01-24T14:19:08.899 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-01-24T14:19:08.899 INFO:teuthology.orchestra.run.smithi019.stdout:monmaptool for a [v2:172.21.15.19:3300,v1:172.21.15.19:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-01-24T14:19:08.899 INFO:teuthology.orchestra.run.smithi019.stdout:setting min_mon_release = pacific 2024-01-24T14:19:08.899 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/monmaptool: set fsid to 654b0c8e-bac3-11ee-95b1-87774f69a715 2024-01-24T14:19:08.899 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-01-24T14:19:08.899 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:19:08.899 INFO:teuthology.orchestra.run.smithi019.stdout:Creating mon... 2024-01-24T14:19:09.531 INFO:teuthology.orchestra.run.smithi019.stdout:create mon.a on 2024-01-24T14:19:09.682 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-01-24T14:19:09.785 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-01-24T14:19:09.954 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-654b0c8e-bac3-11ee-95b1-87774f69a715.target → /etc/systemd/system/ceph-654b0c8e-bac3-11ee-95b1-87774f69a715.target. 2024-01-24T14:19:09.954 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-654b0c8e-bac3-11ee-95b1-87774f69a715.target → /etc/systemd/system/ceph-654b0c8e-bac3-11ee-95b1-87774f69a715.target. 2024-01-24T14:19:10.164 INFO:teuthology.orchestra.run.smithi019.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mon.a 2024-01-24T14:19:10.164 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stderr Failed to reset failed state of unit ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mon.a.service: Unit ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mon.a.service not loaded. 2024-01-24T14:19:10.264 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-654b0c8e-bac3-11ee-95b1-87774f69a715.target.wants/ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mon.a.service → /etc/systemd/system/ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@.service. 2024-01-24T14:19:10.673 INFO:teuthology.orchestra.run.smithi019.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-01-24T14:19:10.673 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stdout disabled 2024-01-24T14:19:10.682 INFO:teuthology.orchestra.run.smithi019.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-01-24T14:19:10.682 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stdout inactive 2024-01-24T14:19:10.682 INFO:teuthology.orchestra.run.smithi019.stdout:firewalld.service is not enabled 2024-01-24T14:19:10.682 INFO:teuthology.orchestra.run.smithi019.stdout:Not possible to enable service . firewalld.service is not available 2024-01-24T14:19:10.682 INFO:teuthology.orchestra.run.smithi019.stdout:Waiting for mon to start... 2024-01-24T14:19:10.682 INFO:teuthology.orchestra.run.smithi019.stdout:Waiting for mon... 2024-01-24T14:19:10.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:10 smithi019 ceph-mon[40169]: mon.a is new leader, mons a in quorum (ranks 0) 2024-01-24T14:19:11.534 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout cluster: 2024-01-24T14:19:11.534 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout id: 654b0c8e-bac3-11ee-95b1-87774f69a715 2024-01-24T14:19:11.534 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-01-24T14:19:11.534 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout 2024-01-24T14:19:11.534 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout services: 2024-01-24T14:19:11.534 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.522401s) 2024-01-24T14:19:11.534 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-01-24T14:19:11.534 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-01-24T14:19:11.534 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout 2024-01-24T14:19:11.535 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout data: 2024-01-24T14:19:11.535 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-01-24T14:19:11.535 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-01-24T14:19:11.535 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-01-24T14:19:11.535 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout pgs: 2024-01-24T14:19:11.535 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout 2024-01-24T14:19:11.535 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.134+0000 7f8da0262700 1 Processor -- start 2024-01-24T14:19:11.535 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.135+0000 7f8da0262700 1 -- start start 2024-01-24T14:19:11.535 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.135+0000 7f8da0262700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8d9807bee0 0x7f8d9807a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:11.535 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.135+0000 7f8da0262700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9807a910 con 0x7f8d9807bee0 2024-01-24T14:19:11.535 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.136+0000 7f8d9dffe700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8d9807bee0 0x7f8d9807a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:11.535 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.136+0000 7f8d9dffe700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8d9807bee0 0x7f8d9807a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:57092/0 (socket says 172.21.15.19:57092) 2024-01-24T14:19:11.535 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.136+0000 7f8d9dffe700 1 -- 172.21.15.19:0/157687632 learned_addr learned my addr 172.21.15.19:0/157687632 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:19:11.535 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.136+0000 7f8d9dffe700 1 -- 172.21.15.19:0/157687632 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d9807aa50 con 0x7f8d9807bee0 2024-01-24T14:19:11.536 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.136+0000 7f8d9dffe700 1 --2- 172.21.15.19:0/157687632 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8d9807bee0 0x7f8d9807a340 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f8d940098a0 tx=0x7f8d94009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f96046a4d9feaca1 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:11.536 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.136+0000 7f8d9cffc700 1 -- 172.21.15.19:0/157687632 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8d94004160 con 0x7f8d9807bee0 2024-01-24T14:19:11.536 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.136+0000 7f8d9cffc700 1 -- 172.21.15.19:0/157687632 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d940042c0 con 0x7f8d9807bee0 2024-01-24T14:19:11.536 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.136+0000 7f8d9cffc700 1 -- 172.21.15.19:0/157687632 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8d940045a0 con 0x7f8d9807bee0 2024-01-24T14:19:11.536 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.137+0000 7f8da0262700 1 -- 172.21.15.19:0/157687632 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8d9807bee0 msgr2=0x7f8d9807a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:11.536 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.137+0000 7f8da0262700 1 --2- 172.21.15.19:0/157687632 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8d9807bee0 0x7f8d9807a340 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f8d940098a0 tx=0x7f8d94009bb0 comp rx=0 tx=0).stop 2024-01-24T14:19:11.536 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.137+0000 7f8da0262700 1 -- 172.21.15.19:0/157687632 shutdown_connections 2024-01-24T14:19:11.536 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.137+0000 7f8da0262700 1 --2- 172.21.15.19:0/157687632 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8d9807bee0 0x7f8d9807a340 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:11.536 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.137+0000 7f8da0262700 1 -- 172.21.15.19:0/157687632 >> 172.21.15.19:0/157687632 conn(0x7f8d98101b60 msgr2=0x7f8d98103f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:19:11.536 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.137+0000 7f8da0262700 1 -- 172.21.15.19:0/157687632 shutdown_connections 2024-01-24T14:19:11.536 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.137+0000 7f8da0262700 1 -- 172.21.15.19:0/157687632 wait complete. 2024-01-24T14:19:11.536 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.138+0000 7f8da0262700 1 Processor -- start 2024-01-24T14:19:11.536 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.138+0000 7f8da0262700 1 -- start start 2024-01-24T14:19:11.537 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.138+0000 7f8da0262700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8d9807bee0 0x7f8d9819bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:11.537 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.138+0000 7f8da0262700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9819c0d0 con 0x7f8d9807bee0 2024-01-24T14:19:11.537 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.138+0000 7f8d9dffe700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8d9807bee0 0x7f8d9819bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:11.537 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.139+0000 7f8d9dffe700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8d9807bee0 0x7f8d9819bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:57094/0 (socket says 172.21.15.19:57094) 2024-01-24T14:19:11.537 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.139+0000 7f8d9dffe700 1 -- 172.21.15.19:0/869853978 learned_addr learned my addr 172.21.15.19:0/869853978 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:19:11.537 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.139+0000 7f8d9dffe700 1 -- 172.21.15.19:0/869853978 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d94009550 con 0x7f8d9807bee0 2024-01-24T14:19:11.537 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.139+0000 7f8d9dffe700 1 --2- 172.21.15.19:0/869853978 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8d9807bee0 0x7f8d9819bb90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8d94004000 tx=0x7f8d94004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:11.537 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.139+0000 7f8d8effd700 1 -- 172.21.15.19:0/869853978 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8d94011630 con 0x7f8d9807bee0 2024-01-24T14:19:11.538 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.139+0000 7f8d8effd700 1 -- 172.21.15.19:0/869853978 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d94011790 con 0x7f8d9807bee0 2024-01-24T14:19:11.538 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.139+0000 7f8da0262700 1 -- 172.21.15.19:0/869853978 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d9819c2d0 con 0x7f8d9807bee0 2024-01-24T14:19:11.538 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.139+0000 7f8d8effd700 1 -- 172.21.15.19:0/869853978 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8d94011a30 con 0x7f8d9807bee0 2024-01-24T14:19:11.538 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.139+0000 7f8da0262700 1 -- 172.21.15.19:0/869853978 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d9819c6f0 con 0x7f8d9807bee0 2024-01-24T14:19:11.538 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.139+0000 7f8d8effd700 1 -- 172.21.15.19:0/869853978 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f8d94022070 con 0x7f8d9807bee0 2024-01-24T14:19:11.538 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.140+0000 7f8d8effd700 1 -- 172.21.15.19:0/869853978 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f8d9401c450 con 0x7f8d9807bee0 2024-01-24T14:19:11.538 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.140+0000 7f8da0262700 1 -- 172.21.15.19:0/869853978 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d98195a80 con 0x7f8d9807bee0 2024-01-24T14:19:11.538 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.143+0000 7f8d8effd700 1 -- 172.21.15.19:0/869853978 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f8d9401b450 con 0x7f8d9807bee0 2024-01-24T14:19:11.538 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.218+0000 7f8da0262700 1 -- 172.21.15.19:0/869853978 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f8d98061950 con 0x7f8d9807bee0 2024-01-24T14:19:11.538 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.219+0000 7f8d8effd700 1 -- 172.21.15.19:0/869853978 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7f8d9401b5f0 con 0x7f8d9807bee0 2024-01-24T14:19:11.538 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.219+0000 7f8da0262700 1 -- 172.21.15.19:0/869853978 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8d9807bee0 msgr2=0x7f8d9819bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:11.538 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.219+0000 7f8da0262700 1 --2- 172.21.15.19:0/869853978 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8d9807bee0 0x7f8d9819bb90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8d94004000 tx=0x7f8d94004750 comp rx=0 tx=0).stop 2024-01-24T14:19:11.538 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.220+0000 7f8da0262700 1 -- 172.21.15.19:0/869853978 shutdown_connections 2024-01-24T14:19:11.539 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.220+0000 7f8da0262700 1 --2- 172.21.15.19:0/869853978 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8d9807bee0 0x7f8d9819bb90 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:11.539 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.220+0000 7f8da0262700 1 -- 172.21.15.19:0/869853978 >> 172.21.15.19:0/869853978 conn(0x7f8d98101b60 msgr2=0x7f8d98103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:19:11.539 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.220+0000 7f8da0262700 1 -- 172.21.15.19:0/869853978 shutdown_connections 2024-01-24T14:19:11.539 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.220+0000 7f8da0262700 1 -- 172.21.15.19:0/869853978 wait complete. 2024-01-24T14:19:11.539 INFO:teuthology.orchestra.run.smithi019.stdout:mon is available 2024-01-24T14:19:11.539 INFO:teuthology.orchestra.run.smithi019.stdout:Assimilating anything we can from ceph.conf... 2024-01-24T14:19:11.985 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:11 smithi019 ceph-mon[40169]: mon.a is new leader, mons a in quorum (ranks 0) 2024-01-24T14:19:11.985 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:11 smithi019 ceph-mon[40169]: monmap e1: 1 mons at {a=[v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-01-24T14:19:11.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:11 smithi019 ceph-mon[40169]: fsmap 2024-01-24T14:19:11.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:11 smithi019 ceph-mon[40169]: osdmap e1: 0 total, 0 up, 0 in 2024-01-24T14:19:11.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:11 smithi019 ceph-mon[40169]: mgrmap e1: no daemons active 2024-01-24T14:19:11.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:11 smithi019 ceph-mon[40169]: from='client.? 172.21.15.19:0/869853978' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-01-24T14:19:12.365 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout 2024-01-24T14:19:12.365 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout [global] 2024-01-24T14:19:12.366 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout fsid = 654b0c8e-bac3-11ee-95b1-87774f69a715 2024-01-24T14:19:12.366 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.19:3300,v1:172.21.15.19:6789] 2024-01-24T14:19:12.366 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-01-24T14:19:12.366 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-01-24T14:19:12.366 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-01-24T14:19:12.366 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-01-24T14:19:12.366 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout 2024-01-24T14:19:12.366 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout [mgr] 2024-01-24T14:19:12.366 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-01-24T14:19:12.366 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout 2024-01-24T14:19:12.366 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout [osd] 2024-01-24T14:19:12.366 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-01-24T14:19:12.366 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-01-24T14:19:12.367 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-01-24T14:19:12.367 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-01-24T14:19:12.367 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-01-24T14:19:12.367 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.934+0000 7f903882c700 1 Processor -- start 2024-01-24T14:19:12.367 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.934+0000 7f903882c700 1 -- start start 2024-01-24T14:19:12.367 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.934+0000 7f903882c700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9034107040 0x7f9034109430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:12.367 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.934+0000 7f903882c700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9034079e70 con 0x7f9034107040 2024-01-24T14:19:12.367 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.935+0000 7f903259c700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9034107040 0x7f9034109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:12.367 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.935+0000 7f903259c700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9034107040 0x7f9034109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:57826/0 (socket says 172.21.15.19:57826) 2024-01-24T14:19:12.367 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.935+0000 7f903259c700 1 -- 172.21.15.19:0/614627642 learned_addr learned my addr 172.21.15.19:0/614627642 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:19:12.367 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.935+0000 7f903259c700 1 -- 172.21.15.19:0/614627642 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9034109970 con 0x7f9034107040 2024-01-24T14:19:12.367 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.935+0000 7f903259c700 1 --2- 172.21.15.19:0/614627642 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9034107040 0x7f9034109430 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f90240098a0 tx=0x7f9024009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a39eaf7b7fd3d6e server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:12.367 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.935+0000 7f903159a700 1 -- 172.21.15.19:0/614627642 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9024004160 con 0x7f9034107040 2024-01-24T14:19:12.367 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.935+0000 7f903159a700 1 -- 172.21.15.19:0/614627642 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90240042c0 con 0x7f9034107040 2024-01-24T14:19:12.368 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.935+0000 7f903159a700 1 -- 172.21.15.19:0/614627642 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f90240045a0 con 0x7f9034107040 2024-01-24T14:19:12.368 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.936+0000 7f903882c700 1 -- 172.21.15.19:0/614627642 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9034107040 msgr2=0x7f9034109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:12.368 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.936+0000 7f903882c700 1 --2- 172.21.15.19:0/614627642 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9034107040 0x7f9034109430 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f90240098a0 tx=0x7f9024009bb0 comp rx=0 tx=0).stop 2024-01-24T14:19:12.368 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.936+0000 7f903882c700 1 -- 172.21.15.19:0/614627642 shutdown_connections 2024-01-24T14:19:12.368 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.936+0000 7f903882c700 1 --2- 172.21.15.19:0/614627642 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9034107040 0x7f9034109430 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:12.368 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.936+0000 7f903882c700 1 -- 172.21.15.19:0/614627642 >> 172.21.15.19:0/614627642 conn(0x7f9034100a70 msgr2=0x7f9034102e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:19:12.368 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.936+0000 7f903882c700 1 -- 172.21.15.19:0/614627642 shutdown_connections 2024-01-24T14:19:12.368 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.936+0000 7f903882c700 1 -- 172.21.15.19:0/614627642 wait complete. 2024-01-24T14:19:12.368 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.937+0000 7f903882c700 1 Processor -- start 2024-01-24T14:19:12.368 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.937+0000 7f903882c700 1 -- start start 2024-01-24T14:19:12.368 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.937+0000 7f903882c700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9034107040 0x7f9034199920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:12.368 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.937+0000 7f903882c700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9034199e60 con 0x7f9034107040 2024-01-24T14:19:12.369 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.937+0000 7f903259c700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9034107040 0x7f9034199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:12.369 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.937+0000 7f903259c700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9034107040 0x7f9034199920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:57832/0 (socket says 172.21.15.19:57832) 2024-01-24T14:19:12.369 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.937+0000 7f903259c700 1 -- 172.21.15.19:0/2044775730 learned_addr learned my addr 172.21.15.19:0/2044775730 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:19:12.369 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.938+0000 7f903259c700 1 -- 172.21.15.19:0/2044775730 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9024009550 con 0x7f9034107040 2024-01-24T14:19:12.369 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.938+0000 7f903259c700 1 --2- 172.21.15.19:0/2044775730 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9034107040 0x7f9034199920 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f90340792b0 tx=0x7f9024004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:12.369 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.938+0000 7f902b7fe700 1 -- 172.21.15.19:0/2044775730 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9024011630 con 0x7f9034107040 2024-01-24T14:19:12.369 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.938+0000 7f902b7fe700 1 -- 172.21.15.19:0/2044775730 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9024011790 con 0x7f9034107040 2024-01-24T14:19:12.369 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.938+0000 7f903882c700 1 -- 172.21.15.19:0/2044775730 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f903419a060 con 0x7f9034107040 2024-01-24T14:19:12.369 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.938+0000 7f902b7fe700 1 -- 172.21.15.19:0/2044775730 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9024011a30 con 0x7f9034107040 2024-01-24T14:19:12.369 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.938+0000 7f903882c700 1 -- 172.21.15.19:0/2044775730 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f903419a480 con 0x7f9034107040 2024-01-24T14:19:12.369 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.938+0000 7f902b7fe700 1 -- 172.21.15.19:0/2044775730 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f9024022070 con 0x7f9034107040 2024-01-24T14:19:12.369 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.938+0000 7f902b7fe700 1 -- 172.21.15.19:0/2044775730 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f902401c450 con 0x7f9034107040 2024-01-24T14:19:12.369 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.939+0000 7f903882c700 1 -- 172.21.15.19:0/2044775730 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9034193890 con 0x7f9034107040 2024-01-24T14:19:12.370 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:11.941+0000 7f902b7fe700 1 -- 172.21.15.19:0/2044775730 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f902401b450 con 0x7f9034107040 2024-01-24T14:19:12.370 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.013+0000 7f903882c700 1 -- 172.21.15.19:0/2044775730 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f903402cea0 con 0x7f9034107040 2024-01-24T14:19:12.370 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.018+0000 7f902b7fe700 1 -- 172.21.15.19:0/2044775730 <== mon.0 v2:172.21.15.19:3300/0 7 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f902401b5f0 con 0x7f9034107040 2024-01-24T14:19:12.370 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.018+0000 7f902b7fe700 1 -- 172.21.15.19:0/2044775730 <== mon.0 v2:172.21.15.19:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7f9024024630 con 0x7f9034107040 2024-01-24T14:19:12.370 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.019+0000 7f903882c700 1 -- 172.21.15.19:0/2044775730 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9034107040 msgr2=0x7f9034199920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:12.370 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.019+0000 7f903882c700 1 --2- 172.21.15.19:0/2044775730 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9034107040 0x7f9034199920 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f90340792b0 tx=0x7f9024004750 comp rx=0 tx=0).stop 2024-01-24T14:19:12.370 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.019+0000 7f903882c700 1 -- 172.21.15.19:0/2044775730 shutdown_connections 2024-01-24T14:19:12.370 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.019+0000 7f903882c700 1 --2- 172.21.15.19:0/2044775730 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9034107040 0x7f9034199920 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:12.370 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.019+0000 7f903882c700 1 -- 172.21.15.19:0/2044775730 >> 172.21.15.19:0/2044775730 conn(0x7f9034100a70 msgr2=0x7f90341022d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:19:12.370 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.019+0000 7f903882c700 1 -- 172.21.15.19:0/2044775730 shutdown_connections 2024-01-24T14:19:12.370 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.019+0000 7f903882c700 1 -- 172.21.15.19:0/2044775730 wait complete. 2024-01-24T14:19:12.370 INFO:teuthology.orchestra.run.smithi019.stdout:Generating new minimal ceph.conf... 2024-01-24T14:19:12.972 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:12 smithi019 ceph-mon[40169]: from='client.? 172.21.15.19:0/2044775730' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-24T14:19:12.973 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:12 smithi019 ceph-mon[40169]: from='client.? 172.21.15.19:0/2044775730' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-01-24T14:19:13.161 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.721+0000 7f467d8cd700 1 Processor -- start 2024-01-24T14:19:13.161 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.721+0000 7f467d8cd700 1 -- start start 2024-01-24T14:19:13.162 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.721+0000 7f467d8cd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f467807be80 0x7f467807a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:13.162 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.721+0000 7f467d8cd700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f467807a8b0 con 0x7f467807be80 2024-01-24T14:19:13.162 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.721+0000 7f4676ffd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f467807be80 0x7f467807a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:13.162 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.721+0000 7f4676ffd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f467807be80 0x7f467807a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:57836/0 (socket says 172.21.15.19:57836) 2024-01-24T14:19:13.162 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.721+0000 7f4676ffd700 1 -- 172.21.15.19:0/2175222913 learned_addr learned my addr 172.21.15.19:0/2175222913 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:19:13.162 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.721+0000 7f4676ffd700 1 -- 172.21.15.19:0/2175222913 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f467807a9f0 con 0x7f467807be80 2024-01-24T14:19:13.162 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.722+0000 7f4676ffd700 1 --2- 172.21.15.19:0/2175222913 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f467807be80 0x7f467807a2e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f46680098a0 tx=0x7f4668009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77813f82aede27b2 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:13.162 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.722+0000 7f46767fc700 1 -- 172.21.15.19:0/2175222913 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4668004160 con 0x7f467807be80 2024-01-24T14:19:13.162 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.722+0000 7f46767fc700 1 -- 172.21.15.19:0/2175222913 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f46680042c0 con 0x7f467807be80 2024-01-24T14:19:13.162 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.722+0000 7f46767fc700 1 -- 172.21.15.19:0/2175222913 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4668011a00 con 0x7f467807be80 2024-01-24T14:19:13.162 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.722+0000 7f467d8cd700 1 -- 172.21.15.19:0/2175222913 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f467807be80 msgr2=0x7f467807a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:13.162 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.722+0000 7f467d8cd700 1 --2- 172.21.15.19:0/2175222913 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f467807be80 0x7f467807a2e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f46680098a0 tx=0x7f4668009bb0 comp rx=0 tx=0).stop 2024-01-24T14:19:13.162 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.722+0000 7f467d8cd700 1 -- 172.21.15.19:0/2175222913 shutdown_connections 2024-01-24T14:19:13.163 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.722+0000 7f467d8cd700 1 --2- 172.21.15.19:0/2175222913 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f467807be80 0x7f467807a2e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:13.163 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.722+0000 7f467d8cd700 1 -- 172.21.15.19:0/2175222913 >> 172.21.15.19:0/2175222913 conn(0x7f4678101b60 msgr2=0x7f4678103f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:19:13.163 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.722+0000 7f467d8cd700 1 -- 172.21.15.19:0/2175222913 shutdown_connections 2024-01-24T14:19:13.163 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.723+0000 7f467d8cd700 1 -- 172.21.15.19:0/2175222913 wait complete. 2024-01-24T14:19:13.163 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.723+0000 7f467d8cd700 1 Processor -- start 2024-01-24T14:19:13.163 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.723+0000 7f467d8cd700 1 -- start start 2024-01-24T14:19:13.163 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.724+0000 7f467d8cd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f467807be80 0x7f467819c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:13.163 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.724+0000 7f467d8cd700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f467819c900 con 0x7f467807be80 2024-01-24T14:19:13.163 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.724+0000 7f4676ffd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f467807be80 0x7f467819c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:13.163 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.724+0000 7f4676ffd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f467807be80 0x7f467819c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:57846/0 (socket says 172.21.15.19:57846) 2024-01-24T14:19:13.163 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.724+0000 7f4676ffd700 1 -- 172.21.15.19:0/1624716916 learned_addr learned my addr 172.21.15.19:0/1624716916 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:19:13.163 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.724+0000 7f4676ffd700 1 -- 172.21.15.19:0/1624716916 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4668009550 con 0x7f467807be80 2024-01-24T14:19:13.163 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.724+0000 7f4676ffd700 1 --2- 172.21.15.19:0/1624716916 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f467807be80 0x7f467819c3c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4668009130 tx=0x7f4668011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:13.163 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.724+0000 7f467c8cb700 1 -- 172.21.15.19:0/1624716916 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4668003b70 con 0x7f467807be80 2024-01-24T14:19:13.164 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.724+0000 7f467c8cb700 1 -- 172.21.15.19:0/1624716916 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4668003cd0 con 0x7f467807be80 2024-01-24T14:19:13.164 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.724+0000 7f467d8cd700 1 -- 172.21.15.19:0/1624716916 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f467819cb00 con 0x7f467807be80 2024-01-24T14:19:13.164 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.724+0000 7f467c8cb700 1 -- 172.21.15.19:0/1624716916 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f46680197d0 con 0x7f467807be80 2024-01-24T14:19:13.164 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.724+0000 7f467d8cd700 1 -- 172.21.15.19:0/1624716916 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f467819cf20 con 0x7f467807be80 2024-01-24T14:19:13.164 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.725+0000 7f467c8cb700 1 -- 172.21.15.19:0/1624716916 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f46680199c0 con 0x7f467807be80 2024-01-24T14:19:13.164 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.725+0000 7f467c8cb700 1 -- 172.21.15.19:0/1624716916 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f466800b720 con 0x7f467807be80 2024-01-24T14:19:13.164 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.726+0000 7f467d8cd700 1 -- 172.21.15.19:0/1624716916 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4678062370 con 0x7f467807be80 2024-01-24T14:19:13.164 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.728+0000 7f467c8cb700 1 -- 172.21.15.19:0/1624716916 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f466802a060 con 0x7f467807be80 2024-01-24T14:19:13.164 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.801+0000 7f467d8cd700 1 -- 172.21.15.19:0/1624716916 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f467819fa50 con 0x7f467807be80 2024-01-24T14:19:13.164 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.801+0000 7f467c8cb700 1 -- 172.21.15.19:0/1624716916 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f4668019e00 con 0x7f467807be80 2024-01-24T14:19:13.164 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.802+0000 7f467d8cd700 1 -- 172.21.15.19:0/1624716916 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f467807be80 msgr2=0x7f467819c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:13.164 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.802+0000 7f467d8cd700 1 --2- 172.21.15.19:0/1624716916 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f467807be80 0x7f467819c3c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4668009130 tx=0x7f4668011ad0 comp rx=0 tx=0).stop 2024-01-24T14:19:13.165 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.802+0000 7f467d8cd700 1 -- 172.21.15.19:0/1624716916 shutdown_connections 2024-01-24T14:19:13.165 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.802+0000 7f467d8cd700 1 --2- 172.21.15.19:0/1624716916 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f467807be80 0x7f467819c3c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:13.165 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.802+0000 7f467d8cd700 1 -- 172.21.15.19:0/1624716916 >> 172.21.15.19:0/1624716916 conn(0x7f4678101b60 msgr2=0x7f46781028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:19:13.165 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.802+0000 7f467d8cd700 1 -- 172.21.15.19:0/1624716916 shutdown_connections 2024-01-24T14:19:13.165 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:12.802+0000 7f467d8cd700 1 -- 172.21.15.19:0/1624716916 wait complete. 2024-01-24T14:19:13.165 INFO:teuthology.orchestra.run.smithi019.stdout:Restarting the monitor... 2024-01-24T14:19:13.235 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:13 smithi019 systemd[1]: Stopping Ceph mon.a for 654b0c8e-bac3-11ee-95b1-87774f69a715... 2024-01-24T14:19:13.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:13 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mon-a[40145]: 2024-01-24T14:19:13.392+0000 7f3508801700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T14:19:13.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:13 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mon-a[40145]: 2024-01-24T14:19:13.392+0000 7f3508801700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-01-24T14:19:13.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:13 smithi019 bash[40505]: ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mon-a 2024-01-24T14:19:14.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 systemd[1]: ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mon.a.service: Succeeded. 2024-01-24T14:19:14.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 systemd[1]: Stopped Ceph mon.a for 654b0c8e-bac3-11ee-95b1-87774f69a715. 2024-01-24T14:19:14.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 systemd[1]: Starting Ceph mon.a for 654b0c8e-bac3-11ee-95b1-87774f69a715... 2024-01-24T14:19:14.326 INFO:teuthology.orchestra.run.smithi019.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-01-24T14:19:14.330 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 bash[40626]: ba9f831db24e5b99026f894d60e3106787143f7b07570b6556f3c9224958e88e 2024-01-24T14:19:14.330 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 systemd[1]: Started Ceph mon.a for 654b0c8e-bac3-11ee-95b1-87774f69a715. 2024-01-24T14:19:14.330 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: set uid:gid to 167:167 (ceph:ceph) 2024-01-24T14:19:14.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable), process ceph-mon, pid 7 2024-01-24T14:19:14.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: pidfile_write: ignore empty --pid-file 2024-01-24T14:19:14.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: load: jerasure load: lrc 2024-01-24T14:19:14.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: RocksDB version: 7.9.2 2024-01-24T14:19:14.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Git sha 0 2024-01-24T14:19:14.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Compile date 2024-01-24 06:39:28 2024-01-24T14:19:14.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: DB SUMMARY 2024-01-24T14:19:14.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: DB Session ID: 6SLIJ062S973E1U76G70 2024-01-24T14:19:14.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: CURRENT file: CURRENT 2024-01-24T14:19:14.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: IDENTITY file: IDENTITY 2024-01-24T14:19:14.738 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-01-24T14:19:14.738 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-01-24T14:19:14.738 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 97550 ; 2024-01-24T14:19:14.738 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.error_if_exists: 0 2024-01-24T14:19:14.738 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.create_if_missing: 0 2024-01-24T14:19:14.738 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.paranoid_checks: 1 2024-01-24T14:19:14.738 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.flush_verify_memtable_count: 1 2024-01-24T14:19:14.738 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-01-24T14:19:14.738 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-01-24T14:19:14.738 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.env: 0x561266115720 2024-01-24T14:19:14.738 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.fs: PosixFileSystem 2024-01-24T14:19:14.738 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.info_log: 0x561267a23460 2024-01-24T14:19:14.738 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.max_file_opening_threads: 16 2024-01-24T14:19:14.739 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.statistics: (nil) 2024-01-24T14:19:14.739 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.use_fsync: 0 2024-01-24T14:19:14.739 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.max_log_file_size: 0 2024-01-24T14:19:14.739 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-24T14:19:14.739 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.log_file_time_to_roll: 0 2024-01-24T14:19:14.739 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.keep_log_file_num: 1000 2024-01-24T14:19:14.739 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.recycle_log_file_num: 0 2024-01-24T14:19:14.739 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.allow_fallocate: 1 2024-01-24T14:19:14.739 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.allow_mmap_reads: 0 2024-01-24T14:19:14.739 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.allow_mmap_writes: 0 2024-01-24T14:19:14.740 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.use_direct_reads: 0 2024-01-24T14:19:14.740 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-24T14:19:14.740 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.create_missing_column_families: 0 2024-01-24T14:19:14.740 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.db_log_dir: 2024-01-24T14:19:14.740 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.wal_dir: 2024-01-24T14:19:14.740 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.table_cache_numshardbits: 6 2024-01-24T14:19:14.740 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.WAL_ttl_seconds: 0 2024-01-24T14:19:14.740 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.WAL_size_limit_MB: 0 2024-01-24T14:19:14.740 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-24T14:19:14.740 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-24T14:19:14.740 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.is_fd_close_on_exec: 1 2024-01-24T14:19:14.740 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.advise_random_on_open: 1 2024-01-24T14:19:14.740 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.db_write_buffer_size: 0 2024-01-24T14:19:14.741 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.write_buffer_manager: 0x561266cb4320 2024-01-24T14:19:14.741 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-24T14:19:14.741 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-24T14:19:14.741 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.use_adaptive_mutex: 0 2024-01-24T14:19:14.741 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.rate_limiter: (nil) 2024-01-24T14:19:14.741 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-24T14:19:14.741 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.wal_recovery_mode: 2 2024-01-24T14:19:14.741 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.enable_thread_tracking: 0 2024-01-24T14:19:14.741 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.enable_pipelined_write: 0 2024-01-24T14:19:14.741 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.unordered_write: 0 2024-01-24T14:19:14.741 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-24T14:19:14.741 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-24T14:19:14.741 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-24T14:19:14.741 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-24T14:19:14.742 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.row_cache: None 2024-01-24T14:19:14.742 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.wal_filter: None 2024-01-24T14:19:14.742 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-24T14:19:14.742 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.allow_ingest_behind: 0 2024-01-24T14:19:14.742 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.two_write_queues: 0 2024-01-24T14:19:14.742 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.manual_wal_flush: 0 2024-01-24T14:19:14.742 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.wal_compression: 0 2024-01-24T14:19:14.742 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.atomic_flush: 0 2024-01-24T14:19:14.742 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-24T14:19:14.742 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.persist_stats_to_disk: 0 2024-01-24T14:19:14.742 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.write_dbid_to_manifest: 0 2024-01-24T14:19:14.742 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.log_readahead_size: 0 2024-01-24T14:19:14.742 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-01-24T14:19:14.742 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.best_efforts_recovery: 0 2024-01-24T14:19:14.742 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-01-24T14:19:14.743 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-01-24T14:19:14.743 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.allow_data_in_errors: 0 2024-01-24T14:19:14.743 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.db_host_id: __hostname__ 2024-01-24T14:19:14.743 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.enforce_single_del_contracts: true 2024-01-24T14:19:14.743 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.max_background_jobs: 2 2024-01-24T14:19:14.743 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.max_background_compactions: -1 2024-01-24T14:19:14.743 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.max_subcompactions: 1 2024-01-24T14:19:14.743 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-24T14:19:14.743 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-24T14:19:14.743 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.delayed_write_rate : 16777216 2024-01-24T14:19:14.743 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.max_total_wal_size: 0 2024-01-24T14:19:14.743 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-24T14:19:14.743 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.stats_dump_period_sec: 600 2024-01-24T14:19:14.743 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.stats_persist_period_sec: 600 2024-01-24T14:19:14.743 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-24T14:19:14.744 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.max_open_files: -1 2024-01-24T14:19:14.744 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.bytes_per_sync: 0 2024-01-24T14:19:14.744 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.wal_bytes_per_sync: 0 2024-01-24T14:19:14.744 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.strict_bytes_per_sync: 0 2024-01-24T14:19:14.744 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.compaction_readahead_size: 0 2024-01-24T14:19:14.744 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.max_background_flushes: -1 2024-01-24T14:19:14.744 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Compression algorithms supported: 2024-01-24T14:19:14.744 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-24T14:19:14.744 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: kZSTD supported: 0 2024-01-24T14:19:14.744 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: kXpressCompression supported: 0 2024-01-24T14:19:14.744 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: kLZ4HCCompression supported: 1 2024-01-24T14:19:14.744 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: kZlibCompression supported: 1 2024-01-24T14:19:14.744 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: kSnappyCompression supported: 1 2024-01-24T14:19:14.745 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: kLZ4Compression supported: 1 2024-01-24T14:19:14.745 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: kBZip2Compression supported: 0 2024-01-24T14:19:14.745 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Fast CRC32 supported: Supported on x86 2024-01-24T14:19:14.745 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: DMutex implementation: pthread_mutex_t 2024-01-24T14:19:14.745 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-01-24T14:19:14.745 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-01-24T14:19:14.745 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-24T14:19:14.745 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.merge_operator: 2024-01-24T14:19:14.745 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.compaction_filter: None 2024-01-24T14:19:14.745 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.compaction_filter_factory: None 2024-01-24T14:19:14.745 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.sst_partitioner_factory: None 2024-01-24T14:19:14.745 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.memtable_factory: SkipListFactory 2024-01-24T14:19:14.745 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.table_factory: BlockBasedTable 2024-01-24T14:19:14.745 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x561267a23560) 2024-01-24T14:19:14.746 INFO:journalctl@ceph.mon.a.smithi019.stdout: cache_index_and_filter_blocks: 1 2024-01-24T14:19:14.746 INFO:journalctl@ceph.mon.a.smithi019.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-24T14:19:14.746 INFO:journalctl@ceph.mon.a.smithi019.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-24T14:19:14.746 INFO:journalctl@ceph.mon.a.smithi019.stdout: pin_top_level_index_and_filter: 1 2024-01-24T14:19:14.746 INFO:journalctl@ceph.mon.a.smithi019.stdout: index_type: 0 2024-01-24T14:19:14.746 INFO:journalctl@ceph.mon.a.smithi019.stdout: data_block_index_type: 0 2024-01-24T14:19:14.746 INFO:journalctl@ceph.mon.a.smithi019.stdout: index_shortening: 1 2024-01-24T14:19:14.746 INFO:journalctl@ceph.mon.a.smithi019.stdout: data_block_hash_table_util_ratio: 0.750000 2024-01-24T14:19:14.746 INFO:journalctl@ceph.mon.a.smithi019.stdout: checksum: 4 2024-01-24T14:19:14.746 INFO:journalctl@ceph.mon.a.smithi019.stdout: no_block_cache: 0 2024-01-24T14:19:14.746 INFO:journalctl@ceph.mon.a.smithi019.stdout: block_cache: 0x561266d351f0 2024-01-24T14:19:14.746 INFO:journalctl@ceph.mon.a.smithi019.stdout: block_cache_name: BinnedLRUCache 2024-01-24T14:19:14.746 INFO:journalctl@ceph.mon.a.smithi019.stdout: block_cache_options: 2024-01-24T14:19:14.746 INFO:journalctl@ceph.mon.a.smithi019.stdout: capacity : 536870912 2024-01-24T14:19:14.746 INFO:journalctl@ceph.mon.a.smithi019.stdout: num_shard_bits : 4 2024-01-24T14:19:14.747 INFO:journalctl@ceph.mon.a.smithi019.stdout: strict_capacity_limit : 0 2024-01-24T14:19:14.747 INFO:journalctl@ceph.mon.a.smithi019.stdout: high_pri_pool_ratio: 0.000 2024-01-24T14:19:14.747 INFO:journalctl@ceph.mon.a.smithi019.stdout: block_cache_compressed: (nil) 2024-01-24T14:19:14.747 INFO:journalctl@ceph.mon.a.smithi019.stdout: persistent_cache: (nil) 2024-01-24T14:19:14.747 INFO:journalctl@ceph.mon.a.smithi019.stdout: block_size: 4096 2024-01-24T14:19:14.747 INFO:journalctl@ceph.mon.a.smithi019.stdout: block_size_deviation: 10 2024-01-24T14:19:14.747 INFO:journalctl@ceph.mon.a.smithi019.stdout: block_restart_interval: 16 2024-01-24T14:19:14.747 INFO:journalctl@ceph.mon.a.smithi019.stdout: index_block_restart_interval: 1 2024-01-24T14:19:14.747 INFO:journalctl@ceph.mon.a.smithi019.stdout: metadata_block_size: 4096 2024-01-24T14:19:14.747 INFO:journalctl@ceph.mon.a.smithi019.stdout: partition_filters: 0 2024-01-24T14:19:14.747 INFO:journalctl@ceph.mon.a.smithi019.stdout: use_delta_encoding: 1 2024-01-24T14:19:14.747 INFO:journalctl@ceph.mon.a.smithi019.stdout: filter_policy: bloomfilter 2024-01-24T14:19:14.747 INFO:journalctl@ceph.mon.a.smithi019.stdout: whole_key_filtering: 1 2024-01-24T14:19:14.748 INFO:journalctl@ceph.mon.a.smithi019.stdout: verify_compression: 0 2024-01-24T14:19:14.748 INFO:journalctl@ceph.mon.a.smithi019.stdout: read_amp_bytes_per_bit: 0 2024-01-24T14:19:14.748 INFO:journalctl@ceph.mon.a.smithi019.stdout: format_version: 5 2024-01-24T14:19:14.748 INFO:journalctl@ceph.mon.a.smithi019.stdout: enable_index_compression: 1 2024-01-24T14:19:14.748 INFO:journalctl@ceph.mon.a.smithi019.stdout: block_align: 0 2024-01-24T14:19:14.748 INFO:journalctl@ceph.mon.a.smithi019.stdout: max_auto_readahead_size: 262144 2024-01-24T14:19:14.748 INFO:journalctl@ceph.mon.a.smithi019.stdout: prepopulate_block_cache: 0 2024-01-24T14:19:14.748 INFO:journalctl@ceph.mon.a.smithi019.stdout: initial_auto_readahead_size: 8192 2024-01-24T14:19:14.748 INFO:journalctl@ceph.mon.a.smithi019.stdout: num_file_reads_for_auto_readahead: 2 2024-01-24T14:19:14.748 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.write_buffer_size: 33554432 2024-01-24T14:19:14.748 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.max_write_buffer_number: 2 2024-01-24T14:19:14.748 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.compression: NoCompression 2024-01-24T14:19:14.748 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.bottommost_compression: Disabled 2024-01-24T14:19:14.749 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.prefix_extractor: nullptr 2024-01-24T14:19:14.749 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-24T14:19:14.749 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.num_levels: 7 2024-01-24T14:19:14.749 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-24T14:19:14.749 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-24T14:19:14.749 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-24T14:19:14.749 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-24T14:19:14.749 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-24T14:19:14.749 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-24T14:19:14.749 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-24T14:19:14.749 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-24T14:19:14.749 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-01-24T14:19:14.749 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-24T14:19:14.749 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-01-24T14:19:14.750 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-01-24T14:19:14.750 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.compression_opts.window_bits: -14 2024-01-24T14:19:14.750 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.compression_opts.level: 32767 2024-01-24T14:19:14.750 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.compression_opts.strategy: 0 2024-01-24T14:19:14.750 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-24T14:19:14.750 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-24T14:19:14.750 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-01-24T14:19:14.750 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-01-24T14:19:14.750 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.compression_opts.enabled: false 2024-01-24T14:19:14.750 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-01-24T14:19:14.750 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-24T14:19:14.750 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-24T14:19:14.750 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-24T14:19:14.750 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.target_file_size_base: 67108864 2024-01-24T14:19:14.750 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.target_file_size_multiplier: 1 2024-01-24T14:19:14.751 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-24T14:19:14.751 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-24T14:19:14.751 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-24T14:19:14.751 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-24T14:19:14.751 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-24T14:19:14.751 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-24T14:19:14.751 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-24T14:19:14.751 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-24T14:19:14.751 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-24T14:19:14.751 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-24T14:19:14.751 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-24T14:19:14.751 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-24T14:19:14.751 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-01-24T14:19:14.751 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.arena_block_size: 1048576 2024-01-24T14:19:14.752 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-24T14:19:14.752 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-24T14:19:14.752 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.disable_auto_compactions: 0 2024-01-24T14:19:14.752 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-24T14:19:14.752 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-24T14:19:14.752 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-24T14:19:14.752 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-24T14:19:14.752 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-24T14:19:14.752 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-24T14:19:14.752 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-24T14:19:14.752 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-24T14:19:14.752 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-24T14:19:14.752 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-24T14:19:14.752 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.table_properties_collectors: 2024-01-24T14:19:14.753 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.inplace_update_support: 0 2024-01-24T14:19:14.753 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.inplace_update_num_locks: 10000 2024-01-24T14:19:14.753 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-24T14:19:14.753 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-24T14:19:14.753 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.memtable_huge_page_size: 0 2024-01-24T14:19:14.753 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.bloom_locality: 0 2024-01-24T14:19:14.753 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.max_successive_merges: 0 2024-01-24T14:19:14.753 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.optimize_filters_for_hits: 0 2024-01-24T14:19:14.753 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.paranoid_file_checks: 0 2024-01-24T14:19:14.753 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.force_consistency_checks: 1 2024-01-24T14:19:14.753 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.report_bg_io_stats: 0 2024-01-24T14:19:14.753 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.ttl: 2592000 2024-01-24T14:19:14.753 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.periodic_compaction_seconds: 0 2024-01-24T14:19:14.753 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-01-24T14:19:14.754 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-01-24T14:19:14.754 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.enable_blob_files: false 2024-01-24T14:19:14.754 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.min_blob_size: 0 2024-01-24T14:19:14.754 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.blob_file_size: 268435456 2024-01-24T14:19:14.754 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.blob_compression_type: NoCompression 2024-01-24T14:19:14.754 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.enable_blob_garbage_collection: false 2024-01-24T14:19:14.754 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-01-24T14:19:14.754 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-01-24T14:19:14.754 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-01-24T14:19:14.754 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.blob_file_starting_level: 0 2024-01-24T14:19:14.754 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-01-24T14:19:14.754 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-01-24T14:19:14.754 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-01-24T14:19:14.754 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: b8eee739-c20c-40ae-9cc9-93f81725b802 2024-01-24T14:19:14.754 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: EVENT_LOG_v1 {"time_micros": 1706105954338392, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-01-24T14:19:14.755 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-01-24T14:19:14.755 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: EVENT_LOG_v1 {"time_micros": 1706105954339744, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 92919, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 307, "table_properties": {"data_size": 90930, "index_size": 271, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13606, "raw_average_key_size": 50, "raw_value_size": 83542, "raw_average_value_size": 309, "num_data_blocks": 11, "num_entries": 270, "num_filter_entries": 270, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1706105954, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "b8eee739-c20c-40ae-9cc9-93f81725b802", "db_session_id": "6SLIJ062S973E1U76G70", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-01-24T14:19:14.755 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: EVENT_LOG_v1 {"time_micros": 1706105954339833, "job": 1, "event": "recovery_finished"} 2024-01-24T14:19:14.755 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-01-24T14:19:14.755 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-01-24T14:19:14.755 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x561266dd2000 2024-01-24T14:19:14.755 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: DB pointer 0x561266dc0000 2024-01-24T14:19:14.755 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-01-24T14:19:14.755 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-01-24T14:19:14.755 INFO:journalctl@ceph.mon.a.smithi019.stdout: ** DB Stats ** 2024-01-24T14:19:14.755 INFO:journalctl@ceph.mon.a.smithi019.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-01-24T14:19:14.755 INFO:journalctl@ceph.mon.a.smithi019.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-24T14:19:14.755 INFO:journalctl@ceph.mon.a.smithi019.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-24T14:19:14.755 INFO:journalctl@ceph.mon.a.smithi019.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-24T14:19:14.756 INFO:journalctl@ceph.mon.a.smithi019.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-24T14:19:14.756 INFO:journalctl@ceph.mon.a.smithi019.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-24T14:19:14.756 INFO:journalctl@ceph.mon.a.smithi019.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-24T14:19:14.756 INFO:journalctl@ceph.mon.a.smithi019.stdout: 2024-01-24T14:19:14.756 INFO:journalctl@ceph.mon.a.smithi019.stdout: ** Compaction Stats [default] ** 2024-01-24T14:19:14.756 INFO:journalctl@ceph.mon.a.smithi019.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-01-24T14:19:14.756 INFO:journalctl@ceph.mon.a.smithi019.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-01-24T14:19:14.756 INFO:journalctl@ceph.mon.a.smithi019.stdout: L0 2/0 92.58 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 103.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-01-24T14:19:14.756 INFO:journalctl@ceph.mon.a.smithi019.stdout: Sum 2/0 92.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 103.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-01-24T14:19:14.756 INFO:journalctl@ceph.mon.a.smithi019.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 103.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-01-24T14:19:14.756 INFO:journalctl@ceph.mon.a.smithi019.stdout: 2024-01-24T14:19:14.756 INFO:journalctl@ceph.mon.a.smithi019.stdout: ** Compaction Stats [default] ** 2024-01-24T14:19:14.756 INFO:journalctl@ceph.mon.a.smithi019.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-01-24T14:19:14.756 INFO:journalctl@ceph.mon.a.smithi019.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-24T14:19:14.756 INFO:journalctl@ceph.mon.a.smithi019.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 103.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-01-24T14:19:14.757 INFO:journalctl@ceph.mon.a.smithi019.stdout: 2024-01-24T14:19:14.757 INFO:journalctl@ceph.mon.a.smithi019.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-01-24T14:19:14.757 INFO:journalctl@ceph.mon.a.smithi019.stdout: 2024-01-24T14:19:14.757 INFO:journalctl@ceph.mon.a.smithi019.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-01-24T14:19:14.757 INFO:journalctl@ceph.mon.a.smithi019.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-01-24T14:19:14.757 INFO:journalctl@ceph.mon.a.smithi019.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-24T14:19:14.757 INFO:journalctl@ceph.mon.a.smithi019.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-01-24T14:19:14.757 INFO:journalctl@ceph.mon.a.smithi019.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-01-24T14:19:14.757 INFO:journalctl@ceph.mon.a.smithi019.stdout: AddFile(Keys): cumulative 0, interval 0 2024-01-24T14:19:14.757 INFO:journalctl@ceph.mon.a.smithi019.stdout: Cumulative compaction: 0.00 GB write, 20.82 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-24T14:19:14.757 INFO:journalctl@ceph.mon.a.smithi019.stdout: Interval compaction: 0.00 GB write, 20.82 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-24T14:19:14.757 INFO:journalctl@ceph.mon.a.smithi019.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-24T14:19:14.757 INFO:journalctl@ceph.mon.a.smithi019.stdout: Block cache BinnedLRUCache@0x561266d351f0#7 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.3e-05 secs_since: 0 2024-01-24T14:19:14.757 INFO:journalctl@ceph.mon.a.smithi019.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-01-24T14:19:14.758 INFO:journalctl@ceph.mon.a.smithi019.stdout: 2024-01-24T14:19:14.758 INFO:journalctl@ceph.mon.a.smithi019.stdout: ** File Read Latency Histogram By Level [default] ** 2024-01-24T14:19:14.758 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: starting mon.a rank 0 at public addrs [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] at bind addrs [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 2024-01-24T14:19:14.758 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: mon.a@-1(???) e1 preinit fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 2024-01-24T14:19:14.758 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: mon.a@-1(???).mds e1 new map 2024-01-24T14:19:14.758 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: mon.a@-1(???).mds e1 print_map 2024-01-24T14:19:14.758 INFO:journalctl@ceph.mon.a.smithi019.stdout: e1 2024-01-24T14:19:14.758 INFO:journalctl@ceph.mon.a.smithi019.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-01-24T14:19:14.758 INFO:journalctl@ceph.mon.a.smithi019.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-24T14:19:14.758 INFO:journalctl@ceph.mon.a.smithi019.stdout: legacy client fscid: -1 2024-01-24T14:19:14.758 INFO:journalctl@ceph.mon.a.smithi019.stdout: 2024-01-24T14:19:14.758 INFO:journalctl@ceph.mon.a.smithi019.stdout: No filesystems configured 2024-01-24T14:19:14.758 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-01-24T14:19:14.758 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-24T14:19:14.758 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-24T14:19:14.759 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-24T14:19:14.759 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-01-24T14:19:14.759 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: mon.a is new leader, mons a in quorum (ranks 0) 2024-01-24T14:19:14.759 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: monmap e1: 1 mons at {a=[v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-01-24T14:19:14.759 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: fsmap 2024-01-24T14:19:14.759 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: osdmap e1: 0 total, 0 up, 0 in 2024-01-24T14:19:14.759 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:14 smithi019 ceph-mon[40663]: mgrmap e1: no daemons active 2024-01-24T14:19:16.840 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:15.651+0000 7fd5968d0700 1 Processor -- start 2024-01-24T14:19:16.840 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:15.651+0000 7fd5968d0700 1 -- start start 2024-01-24T14:19:16.840 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:15.651+0000 7fd5968d0700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd590106580 0x7fd5901069a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:16.840 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:15.651+0000 7fd5968d0700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd590106f70 con 0x7fd590106580 2024-01-24T14:19:16.841 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:15.652+0000 7fd58ffff700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd590106580 0x7fd5901069a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:16.841 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:15.652+0000 7fd58ffff700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd590106580 0x7fd5901069a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:57850/0 (socket says 172.21.15.19:57850) 2024-01-24T14:19:16.841 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:15.652+0000 7fd58ffff700 1 -- 172.21.15.19:0/3969275362 learned_addr learned my addr 172.21.15.19:0/3969275362 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:19:16.841 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:15.652+0000 7fd58ffff700 1 -- 172.21.15.19:0/3969275362 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd590107780 con 0x7fd590106580 2024-01-24T14:19:16.841 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:15.652+0000 7fd58ffff700 1 --2- 172.21.15.19:0/3969275362 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd590106580 0x7fd5901069a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd5800098a0 tx=0x7fd580009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65df8642545245d1 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:16.841 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:15.652+0000 7fd58effd700 1 -- 172.21.15.19:0/3969275362 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd580004160 con 0x7fd590106580 2024-01-24T14:19:16.841 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:15.652+0000 7fd58effd700 1 -- 172.21.15.19:0/3969275362 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd5800042c0 con 0x7fd590106580 2024-01-24T14:19:16.841 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:15.652+0000 7fd58effd700 1 -- 172.21.15.19:0/3969275362 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd580011a00 con 0x7fd590106580 2024-01-24T14:19:16.841 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:15.653+0000 7fd5968d0700 1 -- 172.21.15.19:0/3969275362 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd590106580 msgr2=0x7fd5901069a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:16.841 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:15.653+0000 7fd5968d0700 1 --2- 172.21.15.19:0/3969275362 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd590106580 0x7fd5901069a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd5800098a0 tx=0x7fd580009bb0 comp rx=0 tx=0).stop 2024-01-24T14:19:16.841 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:15.653+0000 7fd5968d0700 1 -- 172.21.15.19:0/3969275362 shutdown_connections 2024-01-24T14:19:16.841 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:15.653+0000 7fd5968d0700 1 --2- 172.21.15.19:0/3969275362 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd590106580 0x7fd5901069a0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:16.841 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:15.653+0000 7fd5968d0700 1 -- 172.21.15.19:0/3969275362 >> 172.21.15.19:0/3969275362 conn(0x7fd590101b60 msgr2=0x7fd590103f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:19:16.841 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:15.653+0000 7fd5968d0700 1 -- 172.21.15.19:0/3969275362 shutdown_connections 2024-01-24T14:19:16.842 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:15.653+0000 7fd5968d0700 1 -- 172.21.15.19:0/3969275362 wait complete. 2024-01-24T14:19:16.842 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:15.654+0000 7fd5968d0700 1 Processor -- start 2024-01-24T14:19:16.842 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:15.654+0000 7fd5968d0700 1 -- start start 2024-01-24T14:19:16.842 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:15.654+0000 7fd5968d0700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd590106580 0x7fd59019a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:16.842 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:15.654+0000 7fd5968d0700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd59019a710 con 0x7fd590106580 2024-01-24T14:19:16.842 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:15.654+0000 7fd58ffff700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd590106580 0x7fd59019a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:16.842 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:15.654+0000 7fd58ffff700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd590106580 0x7fd59019a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:57854/0 (socket says 172.21.15.19:57854) 2024-01-24T14:19:16.842 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:15.654+0000 7fd58ffff700 1 -- 172.21.15.19:0/3688417595 learned_addr learned my addr 172.21.15.19:0/3688417595 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:19:16.842 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:15.655+0000 7fd58ffff700 1 -- 172.21.15.19:0/3688417595 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd580009550 con 0x7fd590106580 2024-01-24T14:19:16.842 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:15.655+0000 7fd58ffff700 1 --2- 172.21.15.19:0/3688417595 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd590106580 0x7fd59019a1d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd580009130 tx=0x7fd580011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:16.842 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:15.655+0000 7fd58d7fa700 1 -- 172.21.15.19:0/3688417595 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd580003ab0 con 0x7fd590106580 2024-01-24T14:19:16.842 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:15.655+0000 7fd58d7fa700 1 -- 172.21.15.19:0/3688417595 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd580003c10 con 0x7fd590106580 2024-01-24T14:19:16.842 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:15.655+0000 7fd58d7fa700 1 -- 172.21.15.19:0/3688417595 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd5800247d0 con 0x7fd590106580 2024-01-24T14:19:16.842 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:15.655+0000 7fd5968d0700 1 -- 172.21.15.19:0/3688417595 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd59019a910 con 0x7fd590106580 2024-01-24T14:19:16.843 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:15.655+0000 7fd5968d0700 1 -- 172.21.15.19:0/3688417595 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd59019ad30 con 0x7fd590106580 2024-01-24T14:19:16.843 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:15.655+0000 7fd58d7fa700 1 -- 172.21.15.19:0/3688417595 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fd580024d00 con 0x7fd590106580 2024-01-24T14:19:16.843 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:15.655+0000 7fd58d7fa700 1 -- 172.21.15.19:0/3688417595 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd5800193c0 con 0x7fd590106580 2024-01-24T14:19:16.843 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:15.656+0000 7fd5968d0700 1 -- 172.21.15.19:0/3688417595 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd590062370 con 0x7fd590106580 2024-01-24T14:19:16.843 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:15.659+0000 7fd58d7fa700 1 -- 172.21.15.19:0/3688417595 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fd58001abb0 con 0x7fd590106580 2024-01-24T14:19:16.843 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:15.730+0000 7fd5968d0700 1 -- 172.21.15.19:0/3688417595 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fd59010a9e0 con 0x7fd590106580 2024-01-24T14:19:16.843 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:15.733+0000 7fd58d7fa700 1 -- 172.21.15.19:0/3688417595 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fd58001ad50 con 0x7fd590106580 2024-01-24T14:19:16.843 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:15.734+0000 7fd5968d0700 1 -- 172.21.15.19:0/3688417595 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd590106580 msgr2=0x7fd59019a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:16.843 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:15.734+0000 7fd5968d0700 1 --2- 172.21.15.19:0/3688417595 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd590106580 0x7fd59019a1d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd580009130 tx=0x7fd580011ad0 comp rx=0 tx=0).stop 2024-01-24T14:19:16.843 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:15.734+0000 7fd5968d0700 1 -- 172.21.15.19:0/3688417595 shutdown_connections 2024-01-24T14:19:16.843 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:15.734+0000 7fd5968d0700 1 --2- 172.21.15.19:0/3688417595 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd590106580 0x7fd59019a1d0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:16.843 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:15.734+0000 7fd5968d0700 1 -- 172.21.15.19:0/3688417595 >> 172.21.15.19:0/3688417595 conn(0x7fd590101b60 msgr2=0x7fd5901028e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:19:16.843 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:15.734+0000 7fd5968d0700 1 -- 172.21.15.19:0/3688417595 shutdown_connections 2024-01-24T14:19:16.844 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:15.734+0000 7fd5968d0700 1 -- 172.21.15.19:0/3688417595 wait complete. 2024-01-24T14:19:16.844 INFO:teuthology.orchestra.run.smithi019.stdout:Wrote config to /etc/ceph/ceph.conf 2024-01-24T14:19:16.964 INFO:teuthology.orchestra.run.smithi019.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-01-24T14:19:16.964 INFO:teuthology.orchestra.run.smithi019.stdout:Creating mgr... 2024-01-24T14:19:16.965 INFO:teuthology.orchestra.run.smithi019.stdout:Verifying port 0.0.0.0:9283 ... 2024-01-24T14:19:16.965 INFO:teuthology.orchestra.run.smithi019.stdout:Verifying port 0.0.0.0:8765 ... 2024-01-24T14:19:16.985 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:16 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/3688417595' entity='client.admin' 2024-01-24T14:19:17.473 INFO:teuthology.orchestra.run.smithi019.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mgr.x 2024-01-24T14:19:17.474 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stderr Failed to reset failed state of unit ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mgr.x.service: Unit ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mgr.x.service not loaded. 2024-01-24T14:19:17.576 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-654b0c8e-bac3-11ee-95b1-87774f69a715.target.wants/ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@.service. 2024-01-24T14:19:17.585 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:17 smithi019 systemd[1]: Starting Ceph mgr.x for 654b0c8e-bac3-11ee-95b1-87774f69a715... 2024-01-24T14:19:19.776 INFO:teuthology.orchestra.run.smithi019.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-01-24T14:19:19.777 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stdout disabled 2024-01-24T14:19:19.784 INFO:teuthology.orchestra.run.smithi019.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-01-24T14:19:19.784 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stdout inactive 2024-01-24T14:19:19.784 INFO:teuthology.orchestra.run.smithi019.stdout:firewalld.service is not enabled 2024-01-24T14:19:19.784 INFO:teuthology.orchestra.run.smithi019.stdout:Not possible to enable service . firewalld.service is not available 2024-01-24T14:19:19.796 INFO:teuthology.orchestra.run.smithi019.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-01-24T14:19:19.797 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stdout disabled 2024-01-24T14:19:19.801 INFO:teuthology.orchestra.run.smithi019.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-01-24T14:19:19.801 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stdout inactive 2024-01-24T14:19:19.802 INFO:teuthology.orchestra.run.smithi019.stdout:firewalld.service is not enabled 2024-01-24T14:19:19.802 INFO:teuthology.orchestra.run.smithi019.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-01-24T14:19:19.802 INFO:teuthology.orchestra.run.smithi019.stdout:Waiting for mgr to start... 2024-01-24T14:19:19.803 INFO:teuthology.orchestra.run.smithi019.stdout:Waiting for mgr... 2024-01-24T14:19:20.111 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:19 smithi019 bash[40899]: 2f53e8cae2b601ee79e6f1c2267d7d2f1cfc73620b1025ee4e14ff7db72bdc3c 2024-01-24T14:19:20.111 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:19 smithi019 systemd[1]: Started Ceph mgr.x for 654b0c8e-bac3-11ee-95b1-87774f69a715. 2024-01-24T14:19:20.405 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:20 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:20.109+0000 7f92ee668200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-24T14:19:20.407 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:20 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:20.213+0000 7f92ee668200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-24T14:19:20.407 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:20 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:20.311+0000 7f92ee668200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-24T14:19:20.407 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:20 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:20.404+0000 7f92ee668200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-24T14:19:21.405 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:21 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/2458533077' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-24T14:19:21.406 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:21 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:21.130+0000 7f92ee668200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-24T14:19:21.406 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:21 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:21.404+0000 7f92ee668200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-24T14:19:21.647 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout 2024-01-24T14:19:21.647 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout { 2024-01-24T14:19:21.647 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "fsid": "654b0c8e-bac3-11ee-95b1-87774f69a715", 2024-01-24T14:19:21.647 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "health": { 2024-01-24T14:19:21.647 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-01-24T14:19:21.648 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-01-24T14:19:21.648 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-01-24T14:19:21.648 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-01-24T14:19:21.648 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-01-24T14:19:21.648 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-01-24T14:19:21.648 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout 0 2024-01-24T14:19:21.648 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout ], 2024-01-24T14:19:21.648 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-01-24T14:19:21.648 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "a" 2024-01-24T14:19:21.648 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout ], 2024-01-24T14:19:21.648 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-01-24T14:19:21.648 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "monmap": { 2024-01-24T14:19:21.649 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-24T14:19:21.649 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-01-24T14:19:21.649 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-01-24T14:19:21.649 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-01-24T14:19:21.649 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-01-24T14:19:21.649 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-24T14:19:21.649 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-01-24T14:19:21.649 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-01-24T14:19:21.649 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-01-24T14:19:21.649 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-01-24T14:19:21.649 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-01-24T14:19:21.649 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-01-24T14:19:21.650 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-01-24T14:19:21.650 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-01-24T14:19:21.650 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-01-24T14:19:21.650 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-01-24T14:19:21.650 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-01-24T14:19:21.650 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-01-24T14:19:21.650 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-01-24T14:19:21.650 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-01-24T14:19:21.650 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-01-24T14:19:21.650 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-01-24T14:19:21.650 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-01-24T14:19:21.650 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-01-24T14:19:21.651 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-24T14:19:21.651 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-01-24T14:19:21.651 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-01-24T14:19:21.651 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-01-24T14:19:21.651 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-01-24T14:19:21.651 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "available": false, 2024-01-24T14:19:21.651 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-01-24T14:19:21.651 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "modules": [ 2024-01-24T14:19:21.651 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "iostat", 2024-01-24T14:19:21.651 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "nfs", 2024-01-24T14:19:21.651 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "restful" 2024-01-24T14:19:21.651 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout ], 2024-01-24T14:19:21.651 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "services": {} 2024-01-24T14:19:21.652 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-01-24T14:19:21.652 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-01-24T14:19:21.652 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-24T14:19:21.652 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "modified": "2024-01-24T14:19:10.700367+0000", 2024-01-24T14:19:21.652 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "services": {} 2024-01-24T14:19:21.652 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-01-24T14:19:21.652 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-01-24T14:19:21.652 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout } 2024-01-24T14:19:21.652 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:21.145+0000 7fd0a92b8700 1 Processor -- start 2024-01-24T14:19:21.652 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:21.145+0000 7fd0a92b8700 1 -- start start 2024-01-24T14:19:21.652 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:21.145+0000 7fd0a92b8700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd0a4106580 0x7fd0a41069a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:21.652 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:21.145+0000 7fd0a92b8700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a4106f70 con 0x7fd0a4106580 2024-01-24T14:19:21.653 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:21.146+0000 7fd0a2ffd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd0a4106580 0x7fd0a41069a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:21.653 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:21.146+0000 7fd0a2ffd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd0a4106580 0x7fd0a41069a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:57880/0 (socket says 172.21.15.19:57880) 2024-01-24T14:19:21.653 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:21.146+0000 7fd0a2ffd700 1 -- 172.21.15.19:0/3579192635 learned_addr learned my addr 172.21.15.19:0/3579192635 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:19:21.653 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:21.146+0000 7fd0a2ffd700 1 -- 172.21.15.19:0/3579192635 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a4107780 con 0x7fd0a4106580 2024-01-24T14:19:21.653 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:21.146+0000 7fd0a2ffd700 1 --2- 172.21.15.19:0/3579192635 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd0a4106580 0x7fd0a41069a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd0940098a0 tx=0x7fd094009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f90176c33a363091 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:21.653 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:21.146+0000 7fd0a1ffb700 1 -- 172.21.15.19:0/3579192635 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd094004160 con 0x7fd0a4106580 2024-01-24T14:19:21.653 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:21.147+0000 7fd0a1ffb700 1 -- 172.21.15.19:0/3579192635 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd0940042c0 con 0x7fd0a4106580 2024-01-24T14:19:21.653 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:21.147+0000 7fd0a1ffb700 1 -- 172.21.15.19:0/3579192635 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd094011a00 con 0x7fd0a4106580 2024-01-24T14:19:21.653 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:21.147+0000 7fd0a92b8700 1 -- 172.21.15.19:0/3579192635 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd0a4106580 msgr2=0x7fd0a41069a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:21.653 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:21.147+0000 7fd0a92b8700 1 --2- 172.21.15.19:0/3579192635 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd0a4106580 0x7fd0a41069a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd0940098a0 tx=0x7fd094009bb0 comp rx=0 tx=0).stop 2024-01-24T14:19:21.653 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:21.147+0000 7fd0a92b8700 1 -- 172.21.15.19:0/3579192635 shutdown_connections 2024-01-24T14:19:21.653 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:21.147+0000 7fd0a92b8700 1 --2- 172.21.15.19:0/3579192635 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd0a4106580 0x7fd0a41069a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:21.653 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:21.147+0000 7fd0a92b8700 1 -- 172.21.15.19:0/3579192635 >> 172.21.15.19:0/3579192635 conn(0x7fd0a4101b60 msgr2=0x7fd0a4103f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:19:21.654 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:21.147+0000 7fd0a92b8700 1 -- 172.21.15.19:0/3579192635 shutdown_connections 2024-01-24T14:19:21.654 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:21.147+0000 7fd0a92b8700 1 -- 172.21.15.19:0/3579192635 wait complete. 2024-01-24T14:19:21.654 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:21.148+0000 7fd0a92b8700 1 Processor -- start 2024-01-24T14:19:21.654 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:21.148+0000 7fd0a92b8700 1 -- start start 2024-01-24T14:19:21.654 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:21.148+0000 7fd0a92b8700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd0a4106580 0x7fd0a419c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:21.654 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:21.148+0000 7fd0a92b8700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a419c940 con 0x7fd0a4106580 2024-01-24T14:19:21.654 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:21.148+0000 7fd0a2ffd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd0a4106580 0x7fd0a419c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:21.654 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:21.149+0000 7fd0a2ffd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd0a4106580 0x7fd0a419c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:57892/0 (socket says 172.21.15.19:57892) 2024-01-24T14:19:21.654 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:21.149+0000 7fd0a2ffd700 1 -- 172.21.15.19:0/2458533077 learned_addr learned my addr 172.21.15.19:0/2458533077 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:19:21.654 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:21.149+0000 7fd0a2ffd700 1 -- 172.21.15.19:0/2458533077 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd094009550 con 0x7fd0a4106580 2024-01-24T14:19:21.654 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:21.149+0000 7fd0a2ffd700 1 --2- 172.21.15.19:0/2458533077 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd0a4106580 0x7fd0a419c400 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd094009fb0 tx=0x7fd094011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:21.654 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:21.149+0000 7fd08bfff700 1 -- 172.21.15.19:0/2458533077 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd094003a40 con 0x7fd0a4106580 2024-01-24T14:19:21.654 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:21.149+0000 7fd08bfff700 1 -- 172.21.15.19:0/2458533077 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd094003ba0 con 0x7fd0a4106580 2024-01-24T14:19:21.655 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:21.149+0000 7fd0a92b8700 1 -- 172.21.15.19:0/2458533077 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0a419cb40 con 0x7fd0a4106580 2024-01-24T14:19:21.655 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:21.149+0000 7fd08bfff700 1 -- 172.21.15.19:0/2458533077 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd09401a750 con 0x7fd0a4106580 2024-01-24T14:19:21.655 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:21.149+0000 7fd0a92b8700 1 -- 172.21.15.19:0/2458533077 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0a419cf60 con 0x7fd0a4106580 2024-01-24T14:19:21.655 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:21.150+0000 7fd08bfff700 1 -- 172.21.15.19:0/2458533077 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fd09401a980 con 0x7fd0a4106580 2024-01-24T14:19:21.655 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:21.150+0000 7fd08bfff700 1 -- 172.21.15.19:0/2458533077 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd09402d9c0 con 0x7fd0a4106580 2024-01-24T14:19:21.655 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:21.150+0000 7fd0a92b8700 1 -- 172.21.15.19:0/2458533077 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0a4062370 con 0x7fd0a4106580 2024-01-24T14:19:21.655 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:21.155+0000 7fd08bfff700 1 -- 172.21.15.19:0/2458533077 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fd09401ad70 con 0x7fd0a4106580 2024-01-24T14:19:21.655 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:21.231+0000 7fd0a92b8700 1 -- 172.21.15.19:0/2458533077 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fd0a410a9b0 con 0x7fd0a4106580 2024-01-24T14:19:21.655 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:21.231+0000 7fd08bfff700 1 -- 172.21.15.19:0/2458533077 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7fd0940248d0 con 0x7fd0a4106580 2024-01-24T14:19:21.655 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:21.232+0000 7fd0a92b8700 1 -- 172.21.15.19:0/2458533077 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd0a4106580 msgr2=0x7fd0a419c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:21.655 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:21.232+0000 7fd0a92b8700 1 --2- 172.21.15.19:0/2458533077 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd0a4106580 0x7fd0a419c400 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd094009fb0 tx=0x7fd094011ad0 comp rx=0 tx=0).stop 2024-01-24T14:19:21.655 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:21.232+0000 7fd0a92b8700 1 -- 172.21.15.19:0/2458533077 shutdown_connections 2024-01-24T14:19:21.656 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:21.232+0000 7fd0a92b8700 1 --2- 172.21.15.19:0/2458533077 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd0a4106580 0x7fd0a419c400 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:21.656 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:21.232+0000 7fd0a92b8700 1 -- 172.21.15.19:0/2458533077 >> 172.21.15.19:0/2458533077 conn(0x7fd0a4101b60 msgr2=0x7fd0a4102910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:19:21.656 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:21.232+0000 7fd0a92b8700 1 -- 172.21.15.19:0/2458533077 shutdown_connections 2024-01-24T14:19:21.656 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:21.232+0000 7fd0a92b8700 1 -- 172.21.15.19:0/2458533077 wait complete. 2024-01-24T14:19:21.656 INFO:teuthology.orchestra.run.smithi019.stdout:mgr not available, waiting (1/15)... 2024-01-24T14:19:21.735 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:21 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:21.501+0000 7f92ee668200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-24T14:19:21.735 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:21 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:21.623+0000 7f92ee668200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-24T14:19:22.235 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:21 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:21.805+0000 7f92ee668200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-24T14:19:22.235 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:21 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:21.949+0000 7f92ee668200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-24T14:19:22.235 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:22 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:22.044+0000 7f92ee668200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-24T14:19:22.735 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:22 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:22.467+0000 7f92ee668200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-24T14:19:23.085 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:22 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:22.763+0000 7f92ee668200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-24T14:19:23.485 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:23 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:23.084+0000 7f92ee668200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-24T14:19:23.485 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:23 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:23.188+0000 7f92ee668200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-24T14:19:24.353 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:24 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/1857637777' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-24T14:19:24.353 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:24 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:24.060+0000 7f92ee668200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-24T14:19:24.353 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:24 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:24.176+0000 7f92ee668200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-24T14:19:24.353 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:24 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:24.351+0000 7f92ee668200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-24T14:19:24.432 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout 2024-01-24T14:19:24.432 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout { 2024-01-24T14:19:24.432 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "fsid": "654b0c8e-bac3-11ee-95b1-87774f69a715", 2024-01-24T14:19:24.432 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "health": { 2024-01-24T14:19:24.432 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-01-24T14:19:24.432 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-01-24T14:19:24.432 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-01-24T14:19:24.433 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-01-24T14:19:24.433 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-01-24T14:19:24.433 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-01-24T14:19:24.433 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout 0 2024-01-24T14:19:24.433 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout ], 2024-01-24T14:19:24.433 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-01-24T14:19:24.433 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "a" 2024-01-24T14:19:24.433 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout ], 2024-01-24T14:19:24.433 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-01-24T14:19:24.433 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "monmap": { 2024-01-24T14:19:24.433 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-24T14:19:24.433 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-01-24T14:19:24.433 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-01-24T14:19:24.433 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-01-24T14:19:24.434 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-01-24T14:19:24.434 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-24T14:19:24.434 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-01-24T14:19:24.434 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-01-24T14:19:24.434 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-01-24T14:19:24.434 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-01-24T14:19:24.434 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-01-24T14:19:24.434 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-01-24T14:19:24.434 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-01-24T14:19:24.434 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-01-24T14:19:24.434 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-01-24T14:19:24.434 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-01-24T14:19:24.434 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-01-24T14:19:24.434 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-01-24T14:19:24.434 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-01-24T14:19:24.435 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-01-24T14:19:24.435 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-01-24T14:19:24.435 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-01-24T14:19:24.435 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-01-24T14:19:24.435 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-01-24T14:19:24.435 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-24T14:19:24.435 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-01-24T14:19:24.435 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-01-24T14:19:24.435 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-01-24T14:19:24.435 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-01-24T14:19:24.435 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "available": false, 2024-01-24T14:19:24.435 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-01-24T14:19:24.435 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "modules": [ 2024-01-24T14:19:24.435 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "iostat", 2024-01-24T14:19:24.435 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "nfs", 2024-01-24T14:19:24.436 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "restful" 2024-01-24T14:19:24.436 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout ], 2024-01-24T14:19:24.436 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "services": {} 2024-01-24T14:19:24.436 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-01-24T14:19:24.436 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-01-24T14:19:24.436 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-24T14:19:24.436 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "modified": "2024-01-24T14:19:10.700367+0000", 2024-01-24T14:19:24.436 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "services": {} 2024-01-24T14:19:24.436 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-01-24T14:19:24.436 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-01-24T14:19:24.436 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout } 2024-01-24T14:19:24.436 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:23.990+0000 7f8b9d143700 1 Processor -- start 2024-01-24T14:19:24.436 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:23.990+0000 7f8b9d143700 1 -- start start 2024-01-24T14:19:24.437 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:23.990+0000 7f8b9d143700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8b98107000 0x7f8b981093f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:24.437 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:23.990+0000 7f8b9d143700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b98079eb0 con 0x7f8b98107000 2024-01-24T14:19:24.437 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:23.991+0000 7f8b96d9d700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8b98107000 0x7f8b981093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:24.437 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:23.991+0000 7f8b96d9d700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8b98107000 0x7f8b981093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:54170/0 (socket says 172.21.15.19:54170) 2024-01-24T14:19:24.437 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:23.991+0000 7f8b96d9d700 1 -- 172.21.15.19:0/4255660714 learned_addr learned my addr 172.21.15.19:0/4255660714 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:19:24.437 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:23.991+0000 7f8b96d9d700 1 -- 172.21.15.19:0/4255660714 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b98109930 con 0x7f8b98107000 2024-01-24T14:19:24.437 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:23.991+0000 7f8b96d9d700 1 --2- 172.21.15.19:0/4255660714 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8b98107000 0x7f8b981093f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8b880098a0 tx=0x7f8b88009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c8c178490884acb server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:24.437 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:23.991+0000 7f8b95d9b700 1 -- 172.21.15.19:0/4255660714 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8b88004160 con 0x7f8b98107000 2024-01-24T14:19:24.437 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:23.991+0000 7f8b95d9b700 1 -- 172.21.15.19:0/4255660714 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8b880042c0 con 0x7f8b98107000 2024-01-24T14:19:24.437 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:23.991+0000 7f8b95d9b700 1 -- 172.21.15.19:0/4255660714 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8b88011a00 con 0x7f8b98107000 2024-01-24T14:19:24.437 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:23.992+0000 7f8b9d143700 1 -- 172.21.15.19:0/4255660714 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8b98107000 msgr2=0x7f8b981093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:24.437 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:23.992+0000 7f8b9d143700 1 --2- 172.21.15.19:0/4255660714 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8b98107000 0x7f8b981093f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8b880098a0 tx=0x7f8b88009bb0 comp rx=0 tx=0).stop 2024-01-24T14:19:24.437 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:23.992+0000 7f8b9d143700 1 -- 172.21.15.19:0/4255660714 shutdown_connections 2024-01-24T14:19:24.437 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:23.992+0000 7f8b9d143700 1 --2- 172.21.15.19:0/4255660714 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8b98107000 0x7f8b981093f0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:24.438 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:23.992+0000 7f8b9d143700 1 -- 172.21.15.19:0/4255660714 >> 172.21.15.19:0/4255660714 conn(0x7f8b98100a70 msgr2=0x7f8b98102e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:19:24.438 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:23.992+0000 7f8b9d143700 1 -- 172.21.15.19:0/4255660714 shutdown_connections 2024-01-24T14:19:24.438 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:23.992+0000 7f8b9d143700 1 -- 172.21.15.19:0/4255660714 wait complete. 2024-01-24T14:19:24.438 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:23.993+0000 7f8b9d143700 1 Processor -- start 2024-01-24T14:19:24.438 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:23.993+0000 7f8b9d143700 1 -- start start 2024-01-24T14:19:24.438 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:23.993+0000 7f8b9d143700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8b98107000 0x7f8b98106960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:24.438 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:23.994+0000 7f8b9d143700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b98106ea0 con 0x7f8b98107000 2024-01-24T14:19:24.438 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:23.994+0000 7f8b96d9d700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8b98107000 0x7f8b98106960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:24.438 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:23.994+0000 7f8b96d9d700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8b98107000 0x7f8b98106960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:54172/0 (socket says 172.21.15.19:54172) 2024-01-24T14:19:24.438 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:23.994+0000 7f8b96d9d700 1 -- 172.21.15.19:0/1857637777 learned_addr learned my addr 172.21.15.19:0/1857637777 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:19:24.438 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:23.994+0000 7f8b96d9d700 1 -- 172.21.15.19:0/1857637777 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b88009550 con 0x7f8b98107000 2024-01-24T14:19:24.438 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:23.994+0000 7f8b96d9d700 1 --2- 172.21.15.19:0/1857637777 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8b98107000 0x7f8b98106960 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8b88009130 tx=0x7f8b88011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:24.438 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:23.994+0000 7f8b7ffff700 1 -- 172.21.15.19:0/1857637777 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8b88003b70 con 0x7f8b98107000 2024-01-24T14:19:24.439 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:23.994+0000 7f8b9d143700 1 -- 172.21.15.19:0/1857637777 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b98104fb0 con 0x7f8b98107000 2024-01-24T14:19:24.439 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:23.994+0000 7f8b7ffff700 1 -- 172.21.15.19:0/1857637777 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8b88003cd0 con 0x7f8b98107000 2024-01-24T14:19:24.439 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:23.994+0000 7f8b7ffff700 1 -- 172.21.15.19:0/1857637777 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8b880197d0 con 0x7f8b98107000 2024-01-24T14:19:24.439 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:23.994+0000 7f8b9d143700 1 -- 172.21.15.19:0/1857637777 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b981053d0 con 0x7f8b98107000 2024-01-24T14:19:24.439 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:23.995+0000 7f8b7ffff700 1 -- 172.21.15.19:0/1857637777 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f8b88019a30 con 0x7f8b98107000 2024-01-24T14:19:24.439 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:23.995+0000 7f8b7ffff700 1 -- 172.21.15.19:0/1857637777 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f8b8800b720 con 0x7f8b98107000 2024-01-24T14:19:24.439 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:23.996+0000 7f8b9d143700 1 -- 172.21.15.19:0/1857637777 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b9810a730 con 0x7f8b98107000 2024-01-24T14:19:24.439 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:23.998+0000 7f8b7ffff700 1 -- 172.21.15.19:0/1857637777 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f8b8802a060 con 0x7f8b98107000 2024-01-24T14:19:24.439 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:24.074+0000 7f8b9d143700 1 -- 172.21.15.19:0/1857637777 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f8b98105a30 con 0x7f8b98107000 2024-01-24T14:19:24.439 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:24.074+0000 7f8b7ffff700 1 -- 172.21.15.19:0/1857637777 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f8b8801abb0 con 0x7f8b98107000 2024-01-24T14:19:24.439 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:24.076+0000 7f8b9d143700 1 -- 172.21.15.19:0/1857637777 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8b98107000 msgr2=0x7f8b98106960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:24.440 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:24.076+0000 7f8b9d143700 1 --2- 172.21.15.19:0/1857637777 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8b98107000 0x7f8b98106960 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8b88009130 tx=0x7f8b88011ad0 comp rx=0 tx=0).stop 2024-01-24T14:19:24.440 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:24.076+0000 7f8b9d143700 1 -- 172.21.15.19:0/1857637777 shutdown_connections 2024-01-24T14:19:24.440 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:24.076+0000 7f8b9d143700 1 --2- 172.21.15.19:0/1857637777 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8b98107000 0x7f8b98106960 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:24.440 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:24.076+0000 7f8b9d143700 1 -- 172.21.15.19:0/1857637777 >> 172.21.15.19:0/1857637777 conn(0x7f8b98100a70 msgr2=0x7f8b981017b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:19:24.440 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:24.076+0000 7f8b9d143700 1 -- 172.21.15.19:0/1857637777 shutdown_connections 2024-01-24T14:19:24.440 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:24.076+0000 7f8b9d143700 1 -- 172.21.15.19:0/1857637777 wait complete. 2024-01-24T14:19:24.441 INFO:teuthology.orchestra.run.smithi019.stdout:mgr not available, waiting (2/15)... 2024-01-24T14:19:25.485 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:25 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:25.071+0000 7f92ee668200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-24T14:19:25.485 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:25 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:25.273+0000 7f92ee668200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-24T14:19:25.985 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:25 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:25.488+0000 7f92ee668200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-24T14:19:26.843 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:26 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:26.531+0000 7f92ee668200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-24T14:19:27.161 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:26 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/982240655' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-24T14:19:27.161 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:26 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:26.842+0000 7f92ee668200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-24T14:19:27.161 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:26 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:26.943+0000 7f92ee668200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-24T14:19:27.240 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout 2024-01-24T14:19:27.241 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout { 2024-01-24T14:19:27.241 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "fsid": "654b0c8e-bac3-11ee-95b1-87774f69a715", 2024-01-24T14:19:27.241 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "health": { 2024-01-24T14:19:27.241 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-01-24T14:19:27.241 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-01-24T14:19:27.241 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-01-24T14:19:27.241 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-01-24T14:19:27.241 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-01-24T14:19:27.241 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-01-24T14:19:27.241 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout 0 2024-01-24T14:19:27.241 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout ], 2024-01-24T14:19:27.241 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-01-24T14:19:27.242 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "a" 2024-01-24T14:19:27.242 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout ], 2024-01-24T14:19:27.242 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-01-24T14:19:27.242 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "monmap": { 2024-01-24T14:19:27.242 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-24T14:19:27.242 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-01-24T14:19:27.242 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-01-24T14:19:27.242 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-01-24T14:19:27.242 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-01-24T14:19:27.242 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-24T14:19:27.242 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-01-24T14:19:27.250 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-01-24T14:19:27.250 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-01-24T14:19:27.250 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-01-24T14:19:27.251 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-01-24T14:19:27.251 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-01-24T14:19:27.251 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-01-24T14:19:27.251 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-01-24T14:19:27.251 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-01-24T14:19:27.251 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-01-24T14:19:27.251 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-01-24T14:19:27.251 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-01-24T14:19:27.251 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-01-24T14:19:27.251 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-01-24T14:19:27.251 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-01-24T14:19:27.251 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-01-24T14:19:27.251 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-01-24T14:19:27.251 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-01-24T14:19:27.251 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-24T14:19:27.252 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-01-24T14:19:27.252 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-01-24T14:19:27.252 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-01-24T14:19:27.252 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-01-24T14:19:27.252 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "available": false, 2024-01-24T14:19:27.252 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-01-24T14:19:27.252 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "modules": [ 2024-01-24T14:19:27.252 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "iostat", 2024-01-24T14:19:27.252 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "nfs", 2024-01-24T14:19:27.252 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "restful" 2024-01-24T14:19:27.252 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout ], 2024-01-24T14:19:27.252 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "services": {} 2024-01-24T14:19:27.252 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-01-24T14:19:27.252 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-01-24T14:19:27.253 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-24T14:19:27.253 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "modified": "2024-01-24T14:19:10.700367+0000", 2024-01-24T14:19:27.253 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "services": {} 2024-01-24T14:19:27.253 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-01-24T14:19:27.253 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-01-24T14:19:27.253 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout } 2024-01-24T14:19:27.253 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:26.803+0000 7f9408982700 1 Processor -- start 2024-01-24T14:19:27.253 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:26.803+0000 7f9408982700 1 -- start start 2024-01-24T14:19:27.253 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:26.804+0000 7f9408982700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9404105490 0x7f94041058b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:27.253 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:26.804+0000 7f9408982700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9404105e80 con 0x7f9404105490 2024-01-24T14:19:27.253 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:26.804+0000 7f940259c700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9404105490 0x7f94041058b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:27.253 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:26.804+0000 7f940259c700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9404105490 0x7f94041058b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:54184/0 (socket says 172.21.15.19:54184) 2024-01-24T14:19:27.253 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:26.804+0000 7f940259c700 1 -- 172.21.15.19:0/2507431872 learned_addr learned my addr 172.21.15.19:0/2507431872 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:19:27.253 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:26.804+0000 7f940259c700 1 -- 172.21.15.19:0/2507431872 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9404106690 con 0x7f9404105490 2024-01-24T14:19:27.254 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:26.804+0000 7f940259c700 1 --2- 172.21.15.19:0/2507431872 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9404105490 0x7f94041058b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f93f40098a0 tx=0x7f93f4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a83f4c8b0679faf3 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:27.254 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:26.804+0000 7f9401d9b700 1 -- 172.21.15.19:0/2507431872 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f93f4004160 con 0x7f9404105490 2024-01-24T14:19:27.254 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:26.805+0000 7f9401d9b700 1 -- 172.21.15.19:0/2507431872 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f93f40042c0 con 0x7f9404105490 2024-01-24T14:19:27.254 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:26.805+0000 7f9401d9b700 1 -- 172.21.15.19:0/2507431872 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f93f4011a00 con 0x7f9404105490 2024-01-24T14:19:27.254 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:26.805+0000 7f9408982700 1 -- 172.21.15.19:0/2507431872 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9404105490 msgr2=0x7f94041058b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:27.254 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:26.805+0000 7f9408982700 1 --2- 172.21.15.19:0/2507431872 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9404105490 0x7f94041058b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f93f40098a0 tx=0x7f93f4009bb0 comp rx=0 tx=0).stop 2024-01-24T14:19:27.254 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:26.805+0000 7f9408982700 1 -- 172.21.15.19:0/2507431872 shutdown_connections 2024-01-24T14:19:27.254 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:26.805+0000 7f9408982700 1 --2- 172.21.15.19:0/2507431872 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9404105490 0x7f94041058b0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:27.254 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:26.805+0000 7f9408982700 1 -- 172.21.15.19:0/2507431872 >> 172.21.15.19:0/2507431872 conn(0x7f9404100a70 msgr2=0x7f9404102e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:19:27.254 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:26.805+0000 7f9408982700 1 -- 172.21.15.19:0/2507431872 shutdown_connections 2024-01-24T14:19:27.254 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:26.806+0000 7f9408982700 1 -- 172.21.15.19:0/2507431872 wait complete. 2024-01-24T14:19:27.254 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:26.806+0000 7f9408982700 1 Processor -- start 2024-01-24T14:19:27.254 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:26.807+0000 7f9408982700 1 -- start start 2024-01-24T14:19:27.254 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:26.807+0000 7f9408982700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9404105490 0x7f9404190620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:27.255 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:26.807+0000 7f9408982700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9404190b60 con 0x7f9404105490 2024-01-24T14:19:27.255 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:26.807+0000 7f940259c700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9404105490 0x7f9404190620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:27.255 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:26.807+0000 7f940259c700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9404105490 0x7f9404190620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:54190/0 (socket says 172.21.15.19:54190) 2024-01-24T14:19:27.255 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:26.807+0000 7f940259c700 1 -- 172.21.15.19:0/982240655 learned_addr learned my addr 172.21.15.19:0/982240655 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:19:27.255 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:26.807+0000 7f940259c700 1 -- 172.21.15.19:0/982240655 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93f4009550 con 0x7f9404105490 2024-01-24T14:19:27.255 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:26.808+0000 7f940259c700 1 --2- 172.21.15.19:0/982240655 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9404105490 0x7f9404190620 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f93f4009130 tx=0x7f93f4011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:27.255 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:26.808+0000 7f93fbfff700 1 -- 172.21.15.19:0/982240655 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f93f4003ab0 con 0x7f9404105490 2024-01-24T14:19:27.255 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:26.808+0000 7f93fbfff700 1 -- 172.21.15.19:0/982240655 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f93f4003c10 con 0x7f9404105490 2024-01-24T14:19:27.255 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:26.808+0000 7f9408982700 1 -- 172.21.15.19:0/982240655 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f940418ec70 con 0x7f9404105490 2024-01-24T14:19:27.255 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:26.808+0000 7f93fbfff700 1 -- 172.21.15.19:0/982240655 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f93f40247d0 con 0x7f9404105490 2024-01-24T14:19:27.255 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:26.808+0000 7f9408982700 1 -- 172.21.15.19:0/982240655 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f940418f090 con 0x7f9404105490 2024-01-24T14:19:27.255 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:26.808+0000 7f93fbfff700 1 -- 172.21.15.19:0/982240655 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f93f4024d50 con 0x7f9404105490 2024-01-24T14:19:27.255 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:26.808+0000 7f93fbfff700 1 -- 172.21.15.19:0/982240655 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f93f40194a0 con 0x7f9404105490 2024-01-24T14:19:27.255 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:26.809+0000 7f9408982700 1 -- 172.21.15.19:0/982240655 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9404062370 con 0x7f9404105490 2024-01-24T14:19:27.256 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:26.811+0000 7f93fbfff700 1 -- 172.21.15.19:0/982240655 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f93f401abb0 con 0x7f9404105490 2024-01-24T14:19:27.256 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:26.888+0000 7f9408982700 1 -- 172.21.15.19:0/982240655 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f940418fa90 con 0x7f9404105490 2024-01-24T14:19:27.256 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:26.888+0000 7f93fbfff700 1 -- 172.21.15.19:0/982240655 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f93f4035090 con 0x7f9404105490 2024-01-24T14:19:27.256 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:26.889+0000 7f9408982700 1 -- 172.21.15.19:0/982240655 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9404105490 msgr2=0x7f9404190620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:27.256 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:26.889+0000 7f9408982700 1 --2- 172.21.15.19:0/982240655 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9404105490 0x7f9404190620 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f93f4009130 tx=0x7f93f4011ad0 comp rx=0 tx=0).stop 2024-01-24T14:19:27.256 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:26.889+0000 7f9408982700 1 -- 172.21.15.19:0/982240655 shutdown_connections 2024-01-24T14:19:27.256 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:26.889+0000 7f9408982700 1 --2- 172.21.15.19:0/982240655 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9404105490 0x7f9404190620 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:27.256 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:26.889+0000 7f9408982700 1 -- 172.21.15.19:0/982240655 >> 172.21.15.19:0/982240655 conn(0x7f9404100a70 msgr2=0x7f9404079920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:19:27.256 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:26.890+0000 7f9408982700 1 -- 172.21.15.19:0/982240655 shutdown_connections 2024-01-24T14:19:27.256 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:26.890+0000 7f9408982700 1 -- 172.21.15.19:0/982240655 wait complete. 2024-01-24T14:19:27.256 INFO:teuthology.orchestra.run.smithi019.stdout:mgr not available, waiting (3/15)... 2024-01-24T14:19:27.485 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:27 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:27.160+0000 7f92ee668200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-24T14:19:28.235 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:27 smithi019 ceph-mon[40663]: Activating manager daemon x 2024-01-24T14:19:28.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:27 smithi019 ceph-mon[40663]: mgrmap e2: x(active, starting, since 0.00363545s) 2024-01-24T14:19:28.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:27 smithi019 ceph-mon[40663]: from='mgr.14100 172.21.15.19:0/3205635533' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-24T14:19:28.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:27 smithi019 ceph-mon[40663]: from='mgr.14100 172.21.15.19:0/3205635533' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-24T14:19:28.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:27 smithi019 ceph-mon[40663]: from='mgr.14100 172.21.15.19:0/3205635533' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-24T14:19:28.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:27 smithi019 ceph-mon[40663]: from='mgr.14100 172.21.15.19:0/3205635533' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-01-24T14:19:28.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:27 smithi019 ceph-mon[40663]: from='mgr.14100 172.21.15.19:0/3205635533' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-01-24T14:19:28.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:27 smithi019 ceph-mon[40663]: Manager daemon x is now available 2024-01-24T14:19:28.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:27 smithi019 ceph-mon[40663]: from='mgr.14100 172.21.15.19:0/3205635533' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-01-24T14:19:28.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:27 smithi019 ceph-mon[40663]: from='mgr.14100 172.21.15.19:0/3205635533' entity='mgr.x' 2024-01-24T14:19:28.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:27 smithi019 ceph-mon[40663]: from='mgr.14100 172.21.15.19:0/3205635533' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-01-24T14:19:28.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:27 smithi019 ceph-mon[40663]: from='mgr.14100 172.21.15.19:0/3205635533' entity='mgr.x' 2024-01-24T14:19:28.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:27 smithi019 ceph-mon[40663]: from='mgr.14100 172.21.15.19:0/3205635533' entity='mgr.x' 2024-01-24T14:19:29.435 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:29 smithi019 ceph-mon[40663]: mgrmap e3: x(active, since 1.00743s) 2024-01-24T14:19:30.302 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout 2024-01-24T14:19:30.302 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout { 2024-01-24T14:19:30.302 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "fsid": "654b0c8e-bac3-11ee-95b1-87774f69a715", 2024-01-24T14:19:30.303 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "health": { 2024-01-24T14:19:30.303 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-01-24T14:19:30.303 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-01-24T14:19:30.303 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-01-24T14:19:30.303 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-01-24T14:19:30.303 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-01-24T14:19:30.304 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-01-24T14:19:30.304 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout 0 2024-01-24T14:19:30.304 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout ], 2024-01-24T14:19:30.304 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-01-24T14:19:30.304 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "a" 2024-01-24T14:19:30.304 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout ], 2024-01-24T14:19:30.304 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "quorum_age": 15, 2024-01-24T14:19:30.304 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "monmap": { 2024-01-24T14:19:30.304 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-24T14:19:30.304 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-01-24T14:19:30.304 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-01-24T14:19:30.305 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-01-24T14:19:30.305 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-01-24T14:19:30.305 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-24T14:19:30.305 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-01-24T14:19:30.305 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-01-24T14:19:30.305 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-01-24T14:19:30.305 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-01-24T14:19:30.306 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-01-24T14:19:30.306 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-01-24T14:19:30.306 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-01-24T14:19:30.306 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-01-24T14:19:30.306 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-01-24T14:19:30.306 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-01-24T14:19:30.306 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-01-24T14:19:30.306 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-01-24T14:19:30.306 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-01-24T14:19:30.306 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-01-24T14:19:30.306 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-01-24T14:19:30.307 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-01-24T14:19:30.307 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-01-24T14:19:30.307 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-01-24T14:19:30.307 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-24T14:19:30.307 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-01-24T14:19:30.307 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-01-24T14:19:30.307 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-01-24T14:19:30.307 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-01-24T14:19:30.307 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "available": true, 2024-01-24T14:19:30.307 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-01-24T14:19:30.307 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "modules": [ 2024-01-24T14:19:30.308 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "iostat", 2024-01-24T14:19:30.308 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "nfs", 2024-01-24T14:19:30.308 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "restful" 2024-01-24T14:19:30.308 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout ], 2024-01-24T14:19:30.308 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "services": {} 2024-01-24T14:19:30.308 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-01-24T14:19:30.308 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-01-24T14:19:30.308 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-01-24T14:19:30.308 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "modified": "2024-01-24T14:19:10.700367+0000", 2024-01-24T14:19:30.308 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "services": {} 2024-01-24T14:19:30.308 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout }, 2024-01-24T14:19:30.308 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-01-24T14:19:30.308 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout } 2024-01-24T14:19:30.309 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.640+0000 7fa94e90c700 1 Processor -- start 2024-01-24T14:19:30.309 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.640+0000 7fa94e90c700 1 -- start start 2024-01-24T14:19:30.309 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.641+0000 7fa94e90c700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa948105490 0x7fa9481058b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:30.309 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.641+0000 7fa94e90c700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa948105e80 con 0x7fa948105490 2024-01-24T14:19:30.309 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.641+0000 7fa947fff700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa948105490 0x7fa9481058b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:30.309 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.641+0000 7fa947fff700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa948105490 0x7fa9481058b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:54256/0 (socket says 172.21.15.19:54256) 2024-01-24T14:19:30.309 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.641+0000 7fa947fff700 1 -- 172.21.15.19:0/3011306489 learned_addr learned my addr 172.21.15.19:0/3011306489 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:19:30.309 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.641+0000 7fa947fff700 1 -- 172.21.15.19:0/3011306489 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa948106690 con 0x7fa948105490 2024-01-24T14:19:30.309 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.641+0000 7fa947fff700 1 --2- 172.21.15.19:0/3011306489 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa948105490 0x7fa9481058b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa9380098a0 tx=0x7fa938009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49799ea679586198 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:30.309 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.641+0000 7fa946ffd700 1 -- 172.21.15.19:0/3011306489 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa938004160 con 0x7fa948105490 2024-01-24T14:19:30.309 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.642+0000 7fa946ffd700 1 -- 172.21.15.19:0/3011306489 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa9380042c0 con 0x7fa948105490 2024-01-24T14:19:30.309 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.642+0000 7fa946ffd700 1 -- 172.21.15.19:0/3011306489 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa938011a00 con 0x7fa948105490 2024-01-24T14:19:30.309 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.642+0000 7fa94e90c700 1 -- 172.21.15.19:0/3011306489 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa948105490 msgr2=0x7fa9481058b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:30.310 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.642+0000 7fa94e90c700 1 --2- 172.21.15.19:0/3011306489 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa948105490 0x7fa9481058b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa9380098a0 tx=0x7fa938009bb0 comp rx=0 tx=0).stop 2024-01-24T14:19:30.310 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.642+0000 7fa94e90c700 1 -- 172.21.15.19:0/3011306489 shutdown_connections 2024-01-24T14:19:30.310 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.642+0000 7fa94e90c700 1 --2- 172.21.15.19:0/3011306489 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa948105490 0x7fa9481058b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:30.310 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.642+0000 7fa94e90c700 1 -- 172.21.15.19:0/3011306489 >> 172.21.15.19:0/3011306489 conn(0x7fa948100a70 msgr2=0x7fa948102e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:19:30.310 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.642+0000 7fa94e90c700 1 -- 172.21.15.19:0/3011306489 shutdown_connections 2024-01-24T14:19:30.310 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.642+0000 7fa94e90c700 1 -- 172.21.15.19:0/3011306489 wait complete. 2024-01-24T14:19:30.310 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.643+0000 7fa94e90c700 1 Processor -- start 2024-01-24T14:19:30.310 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.643+0000 7fa94e90c700 1 -- start start 2024-01-24T14:19:30.310 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.643+0000 7fa94e90c700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa948105490 0x7fa94819a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:30.310 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.643+0000 7fa94e90c700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa94819a730 con 0x7fa948105490 2024-01-24T14:19:30.310 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.643+0000 7fa947fff700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa948105490 0x7fa94819a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:30.310 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.644+0000 7fa947fff700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa948105490 0x7fa94819a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:54264/0 (socket says 172.21.15.19:54264) 2024-01-24T14:19:30.311 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.644+0000 7fa947fff700 1 -- 172.21.15.19:0/1678420426 learned_addr learned my addr 172.21.15.19:0/1678420426 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:19:30.311 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.644+0000 7fa947fff700 1 -- 172.21.15.19:0/1678420426 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa938009550 con 0x7fa948105490 2024-01-24T14:19:30.311 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.644+0000 7fa947fff700 1 --2- 172.21.15.19:0/1678420426 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa948105490 0x7fa94819a1f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa938009fb0 tx=0x7fa938011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:30.311 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.644+0000 7fa9457fa700 1 -- 172.21.15.19:0/1678420426 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa938003a40 con 0x7fa948105490 2024-01-24T14:19:30.311 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.644+0000 7fa9457fa700 1 -- 172.21.15.19:0/1678420426 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa938003ba0 con 0x7fa948105490 2024-01-24T14:19:30.311 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.644+0000 7fa94e90c700 1 -- 172.21.15.19:0/1678420426 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa94819a930 con 0x7fa948105490 2024-01-24T14:19:30.311 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.644+0000 7fa9457fa700 1 -- 172.21.15.19:0/1678420426 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa93801a750 con 0x7fa948105490 2024-01-24T14:19:30.311 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.644+0000 7fa94e90c700 1 -- 172.21.15.19:0/1678420426 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa94819ad50 con 0x7fa948105490 2024-01-24T14:19:30.311 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.645+0000 7fa9457fa700 1 -- 172.21.15.19:0/1678420426 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 3) v1 ==== 45151+0+0 (secure 0 0 0) 0x7fa93801a9d0 con 0x7fa948105490 2024-01-24T14:19:30.311 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.645+0000 7fa94e90c700 1 -- 172.21.15.19:0/1678420426 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa948062370 con 0x7fa948105490 2024-01-24T14:19:30.311 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.645+0000 7fa9457fa700 1 --2- 172.21.15.19:0/1678420426 >> [v2:172.21.15.19:6800/2754937713,v1:172.21.15.19:6801/2754937713] conn(0x7fa930038200 0x7fa93003a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:30.311 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.645+0000 7fa9457fa700 1 -- 172.21.15.19:0/1678420426 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa93804bd90 con 0x7fa948105490 2024-01-24T14:19:30.311 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.646+0000 7fa9477fe700 1 --2- 172.21.15.19:0/1678420426 >> [v2:172.21.15.19:6800/2754937713,v1:172.21.15.19:6801/2754937713] conn(0x7fa930038200 0x7fa93003a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:30.311 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.646+0000 7fa9477fe700 1 --2- 172.21.15.19:0/1678420426 >> [v2:172.21.15.19:6800/2754937713,v1:172.21.15.19:6801/2754937713] conn(0x7fa930038200 0x7fa93003a6c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa93c006fd0 tx=0x7fa93c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:30.312 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.651+0000 7fa9457fa700 1 -- 172.21.15.19:0/1678420426 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fa938018db0 con 0x7fa948105490 2024-01-24T14:19:30.312 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.940+0000 7fa94e90c700 1 -- 172.21.15.19:0/1678420426 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fa9481098c0 con 0x7fa948105490 2024-01-24T14:19:30.312 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.941+0000 7fa9457fa700 1 -- 172.21.15.19:0/1678420426 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7fa938018350 con 0x7fa948105490 2024-01-24T14:19:30.312 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.943+0000 7fa94e90c700 1 -- 172.21.15.19:0/1678420426 >> [v2:172.21.15.19:6800/2754937713,v1:172.21.15.19:6801/2754937713] conn(0x7fa930038200 msgr2=0x7fa93003a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:30.312 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.943+0000 7fa94e90c700 1 --2- 172.21.15.19:0/1678420426 >> [v2:172.21.15.19:6800/2754937713,v1:172.21.15.19:6801/2754937713] conn(0x7fa930038200 0x7fa93003a6c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa93c006fd0 tx=0x7fa93c006e40 comp rx=0 tx=0).stop 2024-01-24T14:19:30.312 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.943+0000 7fa94e90c700 1 -- 172.21.15.19:0/1678420426 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa948105490 msgr2=0x7fa94819a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:30.312 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.943+0000 7fa94e90c700 1 --2- 172.21.15.19:0/1678420426 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa948105490 0x7fa94819a1f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa938009fb0 tx=0x7fa938011ad0 comp rx=0 tx=0).stop 2024-01-24T14:19:30.312 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.943+0000 7fa94e90c700 1 -- 172.21.15.19:0/1678420426 shutdown_connections 2024-01-24T14:19:30.312 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.943+0000 7fa94e90c700 1 --2- 172.21.15.19:0/1678420426 >> [v2:172.21.15.19:6800/2754937713,v1:172.21.15.19:6801/2754937713] conn(0x7fa930038200 0x7fa93003a6c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:30.312 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.943+0000 7fa94e90c700 1 --2- 172.21.15.19:0/1678420426 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa948105490 0x7fa94819a1f0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:30.312 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.943+0000 7fa94e90c700 1 -- 172.21.15.19:0/1678420426 >> 172.21.15.19:0/1678420426 conn(0x7fa948100a70 msgr2=0x7fa948079960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:19:30.312 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.943+0000 7fa94e90c700 1 -- 172.21.15.19:0/1678420426 shutdown_connections 2024-01-24T14:19:30.313 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:29.943+0000 7fa94e90c700 1 -- 172.21.15.19:0/1678420426 wait complete. 2024-01-24T14:19:30.313 INFO:teuthology.orchestra.run.smithi019.stdout:mgr is available 2024-01-24T14:19:30.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:30 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/1678420426' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-24T14:19:30.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:30 smithi019 ceph-mon[40663]: mgrmap e4: x(active, since 3s) 2024-01-24T14:19:31.245 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout 2024-01-24T14:19:31.245 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout [global] 2024-01-24T14:19:31.245 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout fsid = 654b0c8e-bac3-11ee-95b1-87774f69a715 2024-01-24T14:19:31.245 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-01-24T14:19:31.245 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-01-24T14:19:31.245 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-01-24T14:19:31.245 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-01-24T14:19:31.245 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout 2024-01-24T14:19:31.245 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout [mgr] 2024-01-24T14:19:31.245 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-01-24T14:19:31.246 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout 2024-01-24T14:19:31.246 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout [osd] 2024-01-24T14:19:31.246 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-01-24T14:19:31.246 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-01-24T14:19:31.246 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-01-24T14:19:31.246 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-01-24T14:19:31.246 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-01-24T14:19:31.246 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.668+0000 7fdecdba2700 1 Processor -- start 2024-01-24T14:19:31.246 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.668+0000 7fdecdba2700 1 -- start start 2024-01-24T14:19:31.246 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.668+0000 7fdecdba2700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdec8106580 0x7fdec81069a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:31.246 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.668+0000 7fdecdba2700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec8106f70 con 0x7fdec8106580 2024-01-24T14:19:31.246 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.668+0000 7fdec77fe700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdec8106580 0x7fdec81069a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:31.246 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.668+0000 7fdec77fe700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdec8106580 0x7fdec81069a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:54272/0 (socket says 172.21.15.19:54272) 2024-01-24T14:19:31.247 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.668+0000 7fdec77fe700 1 -- 172.21.15.19:0/179763020 learned_addr learned my addr 172.21.15.19:0/179763020 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:19:31.247 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.669+0000 7fdec77fe700 1 -- 172.21.15.19:0/179763020 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec8107780 con 0x7fdec8106580 2024-01-24T14:19:31.247 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.669+0000 7fdec77fe700 1 --2- 172.21.15.19:0/179763020 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdec8106580 0x7fdec81069a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fdeb00098a0 tx=0x7fdeb0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a55daaf6bbb5eac server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:31.247 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.669+0000 7fdec67fc700 1 -- 172.21.15.19:0/179763020 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdeb0004160 con 0x7fdec8106580 2024-01-24T14:19:31.247 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.669+0000 7fdec67fc700 1 -- 172.21.15.19:0/179763020 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdeb00042c0 con 0x7fdec8106580 2024-01-24T14:19:31.247 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.669+0000 7fdec67fc700 1 -- 172.21.15.19:0/179763020 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdeb0011a00 con 0x7fdec8106580 2024-01-24T14:19:31.247 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.670+0000 7fdecdba2700 1 -- 172.21.15.19:0/179763020 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdec8106580 msgr2=0x7fdec81069a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:31.247 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.670+0000 7fdecdba2700 1 --2- 172.21.15.19:0/179763020 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdec8106580 0x7fdec81069a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fdeb00098a0 tx=0x7fdeb0009bb0 comp rx=0 tx=0).stop 2024-01-24T14:19:31.247 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.670+0000 7fdecdba2700 1 -- 172.21.15.19:0/179763020 shutdown_connections 2024-01-24T14:19:31.247 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.670+0000 7fdecdba2700 1 --2- 172.21.15.19:0/179763020 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdec8106580 0x7fdec81069a0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:31.247 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.670+0000 7fdecdba2700 1 -- 172.21.15.19:0/179763020 >> 172.21.15.19:0/179763020 conn(0x7fdec8101b60 msgr2=0x7fdec8103f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:19:31.247 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.670+0000 7fdecdba2700 1 -- 172.21.15.19:0/179763020 shutdown_connections 2024-01-24T14:19:31.247 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.670+0000 7fdecdba2700 1 -- 172.21.15.19:0/179763020 wait complete. 2024-01-24T14:19:31.247 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.670+0000 7fdecdba2700 1 Processor -- start 2024-01-24T14:19:31.247 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.671+0000 7fdecdba2700 1 -- start start 2024-01-24T14:19:31.248 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.671+0000 7fdecdba2700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdec8106580 0x7fdec807bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:31.248 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.671+0000 7fdecdba2700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec807c190 con 0x7fdec8106580 2024-01-24T14:19:31.248 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.671+0000 7fdec77fe700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdec8106580 0x7fdec807bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:31.248 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.671+0000 7fdec77fe700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdec8106580 0x7fdec807bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:54274/0 (socket says 172.21.15.19:54274) 2024-01-24T14:19:31.248 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.671+0000 7fdec77fe700 1 -- 172.21.15.19:0/2340325502 learned_addr learned my addr 172.21.15.19:0/2340325502 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:19:31.248 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.671+0000 7fdec77fe700 1 -- 172.21.15.19:0/2340325502 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeb0009550 con 0x7fdec8106580 2024-01-24T14:19:31.248 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.671+0000 7fdec77fe700 1 --2- 172.21.15.19:0/2340325502 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdec8106580 0x7fdec807bc50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fdeb0009fb0 tx=0x7fdeb0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:31.248 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.672+0000 7fdec4ff9700 1 -- 172.21.15.19:0/2340325502 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdeb0003a40 con 0x7fdec8106580 2024-01-24T14:19:31.248 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.672+0000 7fdecdba2700 1 -- 172.21.15.19:0/2340325502 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdec807a2a0 con 0x7fdec8106580 2024-01-24T14:19:31.248 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.672+0000 7fdec4ff9700 1 -- 172.21.15.19:0/2340325502 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdeb0003ba0 con 0x7fdec8106580 2024-01-24T14:19:31.248 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.672+0000 7fdec4ff9700 1 -- 172.21.15.19:0/2340325502 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdeb001a750 con 0x7fdec8106580 2024-01-24T14:19:31.248 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.672+0000 7fdecdba2700 1 -- 172.21.15.19:0/2340325502 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdec807a6c0 con 0x7fdec8106580 2024-01-24T14:19:31.248 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.673+0000 7fdec4ff9700 1 -- 172.21.15.19:0/2340325502 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 4) v1 ==== 45257+0+0 (secure 0 0 0) 0x7fdeb0024b60 con 0x7fdec8106580 2024-01-24T14:19:31.249 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.673+0000 7fdecdba2700 1 -- 172.21.15.19:0/2340325502 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdec8062370 con 0x7fdec8106580 2024-01-24T14:19:31.249 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.673+0000 7fdec4ff9700 1 --2- 172.21.15.19:0/2340325502 >> [v2:172.21.15.19:6800/2754937713,v1:172.21.15.19:6801/2754937713] conn(0x7fdeb4038320 0x7fdeb403a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:31.249 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.673+0000 7fdec4ff9700 1 -- 172.21.15.19:0/2340325502 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fdeb004bde0 con 0x7fdec8106580 2024-01-24T14:19:31.249 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.673+0000 7fdec6ffd700 1 --2- 172.21.15.19:0/2340325502 >> [v2:172.21.15.19:6800/2754937713,v1:172.21.15.19:6801/2754937713] conn(0x7fdeb4038320 0x7fdeb403a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:31.249 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.674+0000 7fdec6ffd700 1 --2- 172.21.15.19:0/2340325502 >> [v2:172.21.15.19:6800/2754937713,v1:172.21.15.19:6801/2754937713] conn(0x7fdeb4038320 0x7fdeb403a7e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdeb8006fd0 tx=0x7fdeb8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:31.249 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.678+0000 7fdec4ff9700 1 -- 172.21.15.19:0/2340325502 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fdeb0018350 con 0x7fdec8106580 2024-01-24T14:19:31.249 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.883+0000 7fdecdba2700 1 -- 172.21.15.19:0/2340325502 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fdec807afd0 con 0x7fdec8106580 2024-01-24T14:19:31.249 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.883+0000 7fdec4ff9700 1 -- 172.21.15.19:0/2340325502 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7fdeb0018350 con 0x7fdec8106580 2024-01-24T14:19:31.249 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.886+0000 7fdecdba2700 1 -- 172.21.15.19:0/2340325502 >> [v2:172.21.15.19:6800/2754937713,v1:172.21.15.19:6801/2754937713] conn(0x7fdeb4038320 msgr2=0x7fdeb403a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:31.249 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.886+0000 7fdecdba2700 1 --2- 172.21.15.19:0/2340325502 >> [v2:172.21.15.19:6800/2754937713,v1:172.21.15.19:6801/2754937713] conn(0x7fdeb4038320 0x7fdeb403a7e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdeb8006fd0 tx=0x7fdeb8006e40 comp rx=0 tx=0).stop 2024-01-24T14:19:31.249 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.886+0000 7fdecdba2700 1 -- 172.21.15.19:0/2340325502 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdec8106580 msgr2=0x7fdec807bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:31.249 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.886+0000 7fdecdba2700 1 --2- 172.21.15.19:0/2340325502 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdec8106580 0x7fdec807bc50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fdeb0009fb0 tx=0x7fdeb0011ad0 comp rx=0 tx=0).stop 2024-01-24T14:19:31.249 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.886+0000 7fdecdba2700 1 -- 172.21.15.19:0/2340325502 shutdown_connections 2024-01-24T14:19:31.250 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.886+0000 7fdecdba2700 1 --2- 172.21.15.19:0/2340325502 >> [v2:172.21.15.19:6800/2754937713,v1:172.21.15.19:6801/2754937713] conn(0x7fdeb4038320 0x7fdeb403a7e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:31.250 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.886+0000 7fdecdba2700 1 --2- 172.21.15.19:0/2340325502 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdec8106580 0x7fdec807bc50 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:31.250 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.886+0000 7fdecdba2700 1 -- 172.21.15.19:0/2340325502 >> 172.21.15.19:0/2340325502 conn(0x7fdec8101b60 msgr2=0x7fdec81028e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:19:31.250 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.886+0000 7fdecdba2700 1 -- 172.21.15.19:0/2340325502 shutdown_connections 2024-01-24T14:19:31.250 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:30.886+0000 7fdecdba2700 1 -- 172.21.15.19:0/2340325502 wait complete. 2024-01-24T14:19:31.250 INFO:teuthology.orchestra.run.smithi019.stdout:Enabling cephadm module... 2024-01-24T14:19:31.485 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:31 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/2340325502' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-24T14:19:32.682 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:32 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/187391759' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-01-24T14:19:32.682 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:32 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: ignoring --setuser ceph since I am not root 2024-01-24T14:19:32.682 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:32 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: ignoring --setgroup ceph since I am not root 2024-01-24T14:19:32.751 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:31.637+0000 7f918673f700 1 Processor -- start 2024-01-24T14:19:32.751 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:31.637+0000 7f918673f700 1 -- start start 2024-01-24T14:19:32.751 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:31.638+0000 7f918673f700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9180106ff0 0x7f91801093e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:32.751 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:31.638+0000 7f918673f700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9180079eb0 con 0x7f9180106ff0 2024-01-24T14:19:32.751 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:31.638+0000 7f917ffff700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9180106ff0 0x7f91801093e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:32.752 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:31.638+0000 7f917ffff700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9180106ff0 0x7f91801093e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:46320/0 (socket says 172.21.15.19:46320) 2024-01-24T14:19:32.752 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:31.638+0000 7f917ffff700 1 -- 172.21.15.19:0/825560590 learned_addr learned my addr 172.21.15.19:0/825560590 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:19:32.752 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:31.638+0000 7f917ffff700 1 -- 172.21.15.19:0/825560590 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9180109920 con 0x7f9180106ff0 2024-01-24T14:19:32.752 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:31.638+0000 7f917ffff700 1 --2- 172.21.15.19:0/825560590 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9180106ff0 0x7f91801093e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f91700098a0 tx=0x7f9170009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ab67b2fb9c99957 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:32.752 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:31.639+0000 7f917effd700 1 -- 172.21.15.19:0/825560590 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9170004160 con 0x7f9180106ff0 2024-01-24T14:19:32.752 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:31.639+0000 7f917effd700 1 -- 172.21.15.19:0/825560590 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f91700042c0 con 0x7f9180106ff0 2024-01-24T14:19:32.752 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:31.639+0000 7f917effd700 1 -- 172.21.15.19:0/825560590 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9170011a00 con 0x7f9180106ff0 2024-01-24T14:19:32.752 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:31.639+0000 7f918673f700 1 -- 172.21.15.19:0/825560590 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9180106ff0 msgr2=0x7f91801093e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:32.752 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:31.639+0000 7f918673f700 1 --2- 172.21.15.19:0/825560590 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9180106ff0 0x7f91801093e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f91700098a0 tx=0x7f9170009bb0 comp rx=0 tx=0).stop 2024-01-24T14:19:32.752 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:31.639+0000 7f918673f700 1 -- 172.21.15.19:0/825560590 shutdown_connections 2024-01-24T14:19:32.752 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:31.639+0000 7f918673f700 1 --2- 172.21.15.19:0/825560590 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9180106ff0 0x7f91801093e0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:32.752 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:31.639+0000 7f918673f700 1 -- 172.21.15.19:0/825560590 >> 172.21.15.19:0/825560590 conn(0x7f9180100a40 msgr2=0x7f9180102e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:19:32.753 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:31.639+0000 7f918673f700 1 -- 172.21.15.19:0/825560590 shutdown_connections 2024-01-24T14:19:32.753 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:31.639+0000 7f918673f700 1 -- 172.21.15.19:0/825560590 wait complete. 2024-01-24T14:19:32.753 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:31.640+0000 7f918673f700 1 Processor -- start 2024-01-24T14:19:32.753 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:31.640+0000 7f918673f700 1 -- start start 2024-01-24T14:19:32.753 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:31.640+0000 7f918673f700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9180106ff0 0x7f918019a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:32.753 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:31.640+0000 7f918673f700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f918019a6c0 con 0x7f9180106ff0 2024-01-24T14:19:32.753 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:31.641+0000 7f917ffff700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9180106ff0 0x7f918019a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:32.753 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:31.641+0000 7f917ffff700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9180106ff0 0x7f918019a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:46334/0 (socket says 172.21.15.19:46334) 2024-01-24T14:19:32.753 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:31.641+0000 7f917ffff700 1 -- 172.21.15.19:0/187391759 learned_addr learned my addr 172.21.15.19:0/187391759 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:19:32.753 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:31.641+0000 7f917ffff700 1 -- 172.21.15.19:0/187391759 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9170009550 con 0x7f9180106ff0 2024-01-24T14:19:32.753 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:31.641+0000 7f917ffff700 1 --2- 172.21.15.19:0/187391759 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9180106ff0 0x7f918019a180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9170009130 tx=0x7f9170011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:32.753 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:31.641+0000 7f917d7fa700 1 -- 172.21.15.19:0/187391759 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9170003b70 con 0x7f9180106ff0 2024-01-24T14:19:32.753 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:31.641+0000 7f918673f700 1 -- 172.21.15.19:0/187391759 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f918019a8c0 con 0x7f9180106ff0 2024-01-24T14:19:32.754 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:31.641+0000 7f917d7fa700 1 -- 172.21.15.19:0/187391759 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9170003cd0 con 0x7f9180106ff0 2024-01-24T14:19:32.754 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:31.641+0000 7f917d7fa700 1 -- 172.21.15.19:0/187391759 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f91700197d0 con 0x7f9180106ff0 2024-01-24T14:19:32.754 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:31.641+0000 7f918673f700 1 -- 172.21.15.19:0/187391759 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f918019ace0 con 0x7f9180106ff0 2024-01-24T14:19:32.754 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:31.642+0000 7f917d7fa700 1 -- 172.21.15.19:0/187391759 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 4) v1 ==== 45257+0+0 (secure 0 0 0) 0x7f9170019a10 con 0x7f9180106ff0 2024-01-24T14:19:32.754 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:31.642+0000 7f918673f700 1 -- 172.21.15.19:0/187391759 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f918010a720 con 0x7f9180106ff0 2024-01-24T14:19:32.754 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:31.642+0000 7f917d7fa700 1 --2- 172.21.15.19:0/187391759 >> [v2:172.21.15.19:6800/2754937713,v1:172.21.15.19:6801/2754937713] conn(0x7f9168038370 0x7f916803a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:32.754 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:31.642+0000 7f917d7fa700 1 -- 172.21.15.19:0/187391759 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f917004c500 con 0x7f9180106ff0 2024-01-24T14:19:32.754 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:31.643+0000 7f917f7fe700 1 --2- 172.21.15.19:0/187391759 >> [v2:172.21.15.19:6800/2754937713,v1:172.21.15.19:6801/2754937713] conn(0x7f9168038370 0x7f916803a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:32.754 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:31.643+0000 7f917f7fe700 1 --2- 172.21.15.19:0/187391759 >> [v2:172.21.15.19:6800/2754937713,v1:172.21.15.19:6801/2754937713] conn(0x7f9168038370 0x7f916803a830 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9174006fd0 tx=0x7f9174006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:32.754 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:31.647+0000 7f917d7fa700 1 -- 172.21.15.19:0/187391759 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f9170018db0 con 0x7f9180106ff0 2024-01-24T14:19:32.754 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:31.913+0000 7f918673f700 1 -- 172.21.15.19:0/187391759 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f918010a910 con 0x7f9180106ff0 2024-01-24T14:19:32.754 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:32.354+0000 7f917d7fa700 1 -- 172.21.15.19:0/187391759 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mgrmap(e 5) v1 ==== 45268+0+0 (secure 0 0 0) 0x7f9170019cc0 con 0x7f9180106ff0 2024-01-24T14:19:32.754 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:32.355+0000 7f917d7fa700 1 -- 172.21.15.19:0/187391759 <== mon.0 v2:172.21.15.19:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f9170023c00 con 0x7f9180106ff0 2024-01-24T14:19:32.754 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:32.358+0000 7f918673f700 1 -- 172.21.15.19:0/187391759 >> [v2:172.21.15.19:6800/2754937713,v1:172.21.15.19:6801/2754937713] conn(0x7f9168038370 msgr2=0x7f916803a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:32.755 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:32.358+0000 7f918673f700 1 --2- 172.21.15.19:0/187391759 >> [v2:172.21.15.19:6800/2754937713,v1:172.21.15.19:6801/2754937713] conn(0x7f9168038370 0x7f916803a830 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9174006fd0 tx=0x7f9174006e40 comp rx=0 tx=0).stop 2024-01-24T14:19:32.755 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:32.358+0000 7f918673f700 1 -- 172.21.15.19:0/187391759 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9180106ff0 msgr2=0x7f918019a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:32.755 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:32.358+0000 7f918673f700 1 --2- 172.21.15.19:0/187391759 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9180106ff0 0x7f918019a180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9170009130 tx=0x7f9170011ad0 comp rx=0 tx=0).stop 2024-01-24T14:19:32.755 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:32.359+0000 7f918673f700 1 -- 172.21.15.19:0/187391759 shutdown_connections 2024-01-24T14:19:32.755 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:32.359+0000 7f918673f700 1 --2- 172.21.15.19:0/187391759 >> [v2:172.21.15.19:6800/2754937713,v1:172.21.15.19:6801/2754937713] conn(0x7f9168038370 0x7f916803a830 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:32.755 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:32.359+0000 7f918673f700 1 --2- 172.21.15.19:0/187391759 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9180106ff0 0x7f918019a180 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:32.755 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:32.359+0000 7f918673f700 1 -- 172.21.15.19:0/187391759 >> 172.21.15.19:0/187391759 conn(0x7f9180100a40 msgr2=0x7f9180101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:19:32.755 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:32.359+0000 7f918673f700 1 -- 172.21.15.19:0/187391759 shutdown_connections 2024-01-24T14:19:32.755 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:32.359+0000 7f918673f700 1 -- 172.21.15.19:0/187391759 wait complete. 2024-01-24T14:19:32.950 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:32 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:32.841+0000 7fddac782200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-24T14:19:33.235 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:32 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:32.950+0000 7fddac782200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-24T14:19:33.235 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:33 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:33.049+0000 7fddac782200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-24T14:19:33.236 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:33 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:33.146+0000 7fddac782200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-24T14:19:33.645 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:33 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/187391759' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-01-24T14:19:33.645 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:33 smithi019 ceph-mon[40663]: mgrmap e5: x(active, since 5s) 2024-01-24T14:19:33.810 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout { 2024-01-24T14:19:33.810 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-01-24T14:19:33.810 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "available": true, 2024-01-24T14:19:33.811 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-01-24T14:19:33.811 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-01-24T14:19:33.811 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout } 2024-01-24T14:19:33.811 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.162+0000 7fc33ef89700 1 Processor -- start 2024-01-24T14:19:33.811 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.162+0000 7fc33ef89700 1 -- start start 2024-01-24T14:19:33.811 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.163+0000 7fc33ef89700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc338107060 0x7fc338109450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:33.811 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.163+0000 7fc33ef89700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc338079e50 con 0x7fc338107060 2024-01-24T14:19:33.811 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.163+0000 7fc33cd25700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc338107060 0x7fc338109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:33.811 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.163+0000 7fc33cd25700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc338107060 0x7fc338109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:46370/0 (socket says 172.21.15.19:46370) 2024-01-24T14:19:33.811 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.163+0000 7fc33cd25700 1 -- 172.21.15.19:0/55460290 learned_addr learned my addr 172.21.15.19:0/55460290 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:19:33.811 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.163+0000 7fc33cd25700 1 -- 172.21.15.19:0/55460290 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc338109990 con 0x7fc338107060 2024-01-24T14:19:33.811 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.163+0000 7fc33cd25700 1 --2- 172.21.15.19:0/55460290 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc338107060 0x7fc338109450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc32c0098a0 tx=0x7fc32c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30c11c0de92dfb5b server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:33.811 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.164+0000 7fc3377fe700 1 -- 172.21.15.19:0/55460290 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc32c004160 con 0x7fc338107060 2024-01-24T14:19:33.812 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.164+0000 7fc3377fe700 1 -- 172.21.15.19:0/55460290 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc32c0042c0 con 0x7fc338107060 2024-01-24T14:19:33.812 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.164+0000 7fc3377fe700 1 -- 172.21.15.19:0/55460290 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc32c011a00 con 0x7fc338107060 2024-01-24T14:19:33.812 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.164+0000 7fc33ef89700 1 -- 172.21.15.19:0/55460290 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc338107060 msgr2=0x7fc338109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:33.812 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.164+0000 7fc33ef89700 1 --2- 172.21.15.19:0/55460290 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc338107060 0x7fc338109450 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc32c0098a0 tx=0x7fc32c009bb0 comp rx=0 tx=0).stop 2024-01-24T14:19:33.812 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.165+0000 7fc33ef89700 1 -- 172.21.15.19:0/55460290 shutdown_connections 2024-01-24T14:19:33.812 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.165+0000 7fc33ef89700 1 --2- 172.21.15.19:0/55460290 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc338107060 0x7fc338109450 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:33.812 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.165+0000 7fc33ef89700 1 -- 172.21.15.19:0/55460290 >> 172.21.15.19:0/55460290 conn(0x7fc338100a70 msgr2=0x7fc338102e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:19:33.812 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.165+0000 7fc33ef89700 1 -- 172.21.15.19:0/55460290 shutdown_connections 2024-01-24T14:19:33.812 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.165+0000 7fc33ef89700 1 -- 172.21.15.19:0/55460290 wait complete. 2024-01-24T14:19:33.812 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.165+0000 7fc33ef89700 1 Processor -- start 2024-01-24T14:19:33.812 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.165+0000 7fc33ef89700 1 -- start start 2024-01-24T14:19:33.812 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.166+0000 7fc33ef89700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc338107060 0x7fc33819a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:33.812 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.166+0000 7fc33ef89700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc33819a700 con 0x7fc338107060 2024-01-24T14:19:33.813 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.166+0000 7fc33cd25700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc338107060 0x7fc33819a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:33.813 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.166+0000 7fc33cd25700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc338107060 0x7fc33819a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:46372/0 (socket says 172.21.15.19:46372) 2024-01-24T14:19:33.813 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.166+0000 7fc33cd25700 1 -- 172.21.15.19:0/628728091 learned_addr learned my addr 172.21.15.19:0/628728091 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:19:33.813 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.166+0000 7fc33cd25700 1 -- 172.21.15.19:0/628728091 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc32c009550 con 0x7fc338107060 2024-01-24T14:19:33.813 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.166+0000 7fc33cd25700 1 --2- 172.21.15.19:0/628728091 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc338107060 0x7fc33819a1c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc32c009fb0 tx=0x7fc32c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:33.813 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.166+0000 7fc335ffb700 1 -- 172.21.15.19:0/628728091 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc32c003a40 con 0x7fc338107060 2024-01-24T14:19:33.813 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.167+0000 7fc33ef89700 1 -- 172.21.15.19:0/628728091 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc33819a900 con 0x7fc338107060 2024-01-24T14:19:33.813 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.167+0000 7fc335ffb700 1 -- 172.21.15.19:0/628728091 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc32c003ba0 con 0x7fc338107060 2024-01-24T14:19:33.813 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.167+0000 7fc335ffb700 1 -- 172.21.15.19:0/628728091 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc32c01a750 con 0x7fc338107060 2024-01-24T14:19:33.813 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.167+0000 7fc33ef89700 1 -- 172.21.15.19:0/628728091 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc33819ad20 con 0x7fc338107060 2024-01-24T14:19:33.813 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.168+0000 7fc335ffb700 1 -- 172.21.15.19:0/628728091 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) v1 ==== 45268+0+0 (secure 0 0 0) 0x7fc32c01a9d0 con 0x7fc338107060 2024-01-24T14:19:33.813 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.168+0000 7fc335ffb700 1 --2- 172.21.15.19:0/628728091 >> [v2:172.21.15.19:6800/2754937713,v1:172.21.15.19:6801/2754937713] conn(0x7fc328038370 0x7fc32803a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:33.813 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.168+0000 7fc335ffb700 1 -- 172.21.15.19:0/628728091 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc32c04bed0 con 0x7fc338107060 2024-01-24T14:19:33.813 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.168+0000 7fc33ef89700 1 -- 172.21.15.19:0/628728091 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc338062370 con 0x7fc338107060 2024-01-24T14:19:33.814 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.168+0000 7fc337fff700 1 -- 172.21.15.19:0/628728091 >> [v2:172.21.15.19:6800/2754937713,v1:172.21.15.19:6801/2754937713] conn(0x7fc328038370 msgr2=0x7fc32803a830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:6800/2754937713 2024-01-24T14:19:33.814 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.168+0000 7fc337fff700 1 --2- 172.21.15.19:0/628728091 >> [v2:172.21.15.19:6800/2754937713,v1:172.21.15.19:6801/2754937713] conn(0x7fc328038370 0x7fc32803a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-24T14:19:33.814 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.173+0000 7fc335ffb700 1 -- 172.21.15.19:0/628728091 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fc32c023390 con 0x7fc338107060 2024-01-24T14:19:33.814 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.369+0000 7fc337fff700 1 -- 172.21.15.19:0/628728091 >> [v2:172.21.15.19:6800/2754937713,v1:172.21.15.19:6801/2754937713] conn(0x7fc328038370 msgr2=0x7fc32803a830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:6800/2754937713 2024-01-24T14:19:33.814 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.369+0000 7fc337fff700 1 --2- 172.21.15.19:0/628728091 >> [v2:172.21.15.19:6800/2754937713,v1:172.21.15.19:6801/2754937713] conn(0x7fc328038370 0x7fc32803a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-24T14:19:33.814 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.444+0000 7fc33ef89700 1 -- 172.21.15.19:0/628728091 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fc33819d760 con 0x7fc338107060 2024-01-24T14:19:33.814 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.444+0000 7fc335ffb700 1 -- 172.21.15.19:0/628728091 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7fc32c018b40 con 0x7fc338107060 2024-01-24T14:19:33.814 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.446+0000 7fc33ef89700 1 -- 172.21.15.19:0/628728091 >> [v2:172.21.15.19:6800/2754937713,v1:172.21.15.19:6801/2754937713] conn(0x7fc328038370 msgr2=0x7fc32803a830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-24T14:19:33.814 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.446+0000 7fc33ef89700 1 --2- 172.21.15.19:0/628728091 >> [v2:172.21.15.19:6800/2754937713,v1:172.21.15.19:6801/2754937713] conn(0x7fc328038370 0x7fc32803a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:33.814 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.446+0000 7fc33ef89700 1 -- 172.21.15.19:0/628728091 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc338107060 msgr2=0x7fc33819a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:33.814 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.446+0000 7fc33ef89700 1 --2- 172.21.15.19:0/628728091 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc338107060 0x7fc33819a1c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc32c009fb0 tx=0x7fc32c011ad0 comp rx=0 tx=0).stop 2024-01-24T14:19:33.814 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.446+0000 7fc33ef89700 1 -- 172.21.15.19:0/628728091 shutdown_connections 2024-01-24T14:19:33.814 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.446+0000 7fc33ef89700 1 --2- 172.21.15.19:0/628728091 >> [v2:172.21.15.19:6800/2754937713,v1:172.21.15.19:6801/2754937713] conn(0x7fc328038370 0x7fc32803a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:33.814 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.446+0000 7fc33ef89700 1 --2- 172.21.15.19:0/628728091 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc338107060 0x7fc33819a1c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:33.815 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.447+0000 7fc33ef89700 1 -- 172.21.15.19:0/628728091 >> 172.21.15.19:0/628728091 conn(0x7fc338100a70 msgr2=0x7fc3381017d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:19:33.815 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.447+0000 7fc33ef89700 1 -- 172.21.15.19:0/628728091 shutdown_connections 2024-01-24T14:19:33.815 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:33.447+0000 7fc33ef89700 1 -- 172.21.15.19:0/628728091 wait complete. 2024-01-24T14:19:33.815 INFO:teuthology.orchestra.run.smithi019.stdout:Waiting for the mgr to restart... 2024-01-24T14:19:33.815 INFO:teuthology.orchestra.run.smithi019.stdout:Waiting for mgr epoch 5... 2024-01-24T14:19:33.912 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:33 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:33.861+0000 7fddac782200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-24T14:19:34.235 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:34 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:34.136+0000 7fddac782200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-24T14:19:34.235 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:34 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:34.234+0000 7fddac782200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-24T14:19:34.537 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:34 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/628728091' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-01-24T14:19:34.538 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:34 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:34.356+0000 7fddac782200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-24T14:19:34.985 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:34 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:34.536+0000 7fddac782200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-24T14:19:34.985 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:34 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:34.680+0000 7fddac782200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-24T14:19:34.986 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:34 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:34.775+0000 7fddac782200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-24T14:19:35.485 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:35 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:35.199+0000 7fddac782200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-24T14:19:35.819 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:35 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:35.496+0000 7fddac782200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-24T14:19:35.819 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:35 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:35.818+0000 7fddac782200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-24T14:19:36.235 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:35 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:35.922+0000 7fddac782200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-24T14:19:37.047 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:36 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:36.774+0000 7fddac782200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-24T14:19:37.047 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:36 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:36.880+0000 7fddac782200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-24T14:19:37.047 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:37 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:37.046+0000 7fddac782200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-24T14:19:38.185 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:37 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:37.762+0000 7fddac782200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-24T14:19:38.185 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:37 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:37.966+0000 7fddac782200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-24T14:19:38.185 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:38 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:38.184+0000 7fddac782200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-24T14:19:39.485 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:39 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:39.223+0000 7fddac782200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-24T14:19:39.840 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:39 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:39.525+0000 7fddac782200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-24T14:19:39.840 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:39 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:39.623+0000 7fddac782200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-24T14:19:39.840 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:39 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:39.839+0000 7fddac782200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-24T14:19:40.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:40 smithi019 ceph-mon[40663]: Active manager daemon x restarted 2024-01-24T14:19:40.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:40 smithi019 ceph-mon[40663]: Activating manager daemon x 2024-01-24T14:19:40.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:40 smithi019 ceph-mon[40663]: osdmap e2: 0 total, 0 up, 0 in 2024-01-24T14:19:40.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:40 smithi019 ceph-mon[40663]: mgrmap e6: x(active, starting, since 0.00360356s) 2024-01-24T14:19:40.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:40 smithi019 ceph-mon[40663]: from='mgr.14120 172.21.15.19:0/1799968416' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-01-24T14:19:40.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:40 smithi019 ceph-mon[40663]: from='mgr.14120 172.21.15.19:0/1799968416' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-01-24T14:19:40.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:40 smithi019 ceph-mon[40663]: from='mgr.14120 172.21.15.19:0/1799968416' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-24T14:19:40.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:40 smithi019 ceph-mon[40663]: from='mgr.14120 172.21.15.19:0/1799968416' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-24T14:19:40.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:40 smithi019 ceph-mon[40663]: from='mgr.14120 172.21.15.19:0/1799968416' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-24T14:19:40.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:40 smithi019 ceph-mon[40663]: Manager daemon x is now available 2024-01-24T14:19:40.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:40 smithi019 ceph-mon[40663]: from='mgr.14120 172.21.15.19:0/1799968416' entity='mgr.x' 2024-01-24T14:19:40.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:40 smithi019 ceph-mon[40663]: from='mgr.14120 172.21.15.19:0/1799968416' entity='mgr.x' 2024-01-24T14:19:40.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:40 smithi019 ceph-mon[40663]: from='mgr.14120 172.21.15.19:0/1799968416' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:19:40.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:40 smithi019 ceph-mon[40663]: from='mgr.14120 172.21.15.19:0/1799968416' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:19:40.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:40 smithi019 ceph-mon[40663]: from='mgr.14120 172.21.15.19:0/1799968416' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:19:40.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:40 smithi019 ceph-mon[40663]: from='mgr.14120 172.21.15.19:0/1799968416' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-01-24T14:19:40.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:40 smithi019 ceph-mon[40663]: from='mgr.14120 172.21.15.19:0/1799968416' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-01-24T14:19:41.260 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout { 2024-01-24T14:19:41.260 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-01-24T14:19:41.260 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "initialized": true 2024-01-24T14:19:41.260 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout } 2024-01-24T14:19:41.260 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:34.163+0000 7fdf89561700 1 Processor -- start 2024-01-24T14:19:41.260 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:34.163+0000 7fdf89561700 1 -- start start 2024-01-24T14:19:41.260 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:34.163+0000 7fdf89561700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdf84104860 0x7fdf84104c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:41.260 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:34.163+0000 7fdf89561700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf84105250 con 0x7fdf84104860 2024-01-24T14:19:41.260 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:34.163+0000 7fdf82ffd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdf84104860 0x7fdf84104c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:41.260 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:34.163+0000 7fdf82ffd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdf84104860 0x7fdf84104c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:46384/0 (socket says 172.21.15.19:46384) 2024-01-24T14:19:41.261 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:34.163+0000 7fdf82ffd700 1 -- 172.21.15.19:0/3565170605 learned_addr learned my addr 172.21.15.19:0/3565170605 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:19:41.261 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:34.164+0000 7fdf82ffd700 1 -- 172.21.15.19:0/3565170605 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf841077f0 con 0x7fdf84104860 2024-01-24T14:19:41.261 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:34.164+0000 7fdf82ffd700 1 --2- 172.21.15.19:0/3565170605 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdf84104860 0x7fdf84104c80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fdf740098a0 tx=0x7fdf74009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecceb3df39cee080 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:41.261 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:34.164+0000 7fdf81ffb700 1 -- 172.21.15.19:0/3565170605 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdf74004160 con 0x7fdf84104860 2024-01-24T14:19:41.261 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:34.164+0000 7fdf81ffb700 1 -- 172.21.15.19:0/3565170605 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdf740042c0 con 0x7fdf84104860 2024-01-24T14:19:41.261 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:34.164+0000 7fdf81ffb700 1 -- 172.21.15.19:0/3565170605 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdf74011a00 con 0x7fdf84104860 2024-01-24T14:19:41.261 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:34.165+0000 7fdf89561700 1 -- 172.21.15.19:0/3565170605 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdf84104860 msgr2=0x7fdf84104c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:41.261 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:34.165+0000 7fdf89561700 1 --2- 172.21.15.19:0/3565170605 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdf84104860 0x7fdf84104c80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fdf740098a0 tx=0x7fdf74009bb0 comp rx=0 tx=0).stop 2024-01-24T14:19:41.261 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:34.165+0000 7fdf89561700 1 -- 172.21.15.19:0/3565170605 shutdown_connections 2024-01-24T14:19:41.261 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:34.165+0000 7fdf89561700 1 --2- 172.21.15.19:0/3565170605 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdf84104860 0x7fdf84104c80 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:41.261 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:34.165+0000 7fdf89561700 1 -- 172.21.15.19:0/3565170605 >> 172.21.15.19:0/3565170605 conn(0x7fdf84100190 msgr2=0x7fdf841025d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:19:41.261 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:34.165+0000 7fdf89561700 1 -- 172.21.15.19:0/3565170605 shutdown_connections 2024-01-24T14:19:41.261 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:34.165+0000 7fdf89561700 1 -- 172.21.15.19:0/3565170605 wait complete. 2024-01-24T14:19:41.261 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:34.165+0000 7fdf89561700 1 Processor -- start 2024-01-24T14:19:41.262 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:34.166+0000 7fdf89561700 1 -- start start 2024-01-24T14:19:41.262 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:34.166+0000 7fdf89561700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdf84104860 0x7fdf8419a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:41.262 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:34.166+0000 7fdf89561700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8419a670 con 0x7fdf84104860 2024-01-24T14:19:41.262 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:34.166+0000 7fdf82ffd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdf84104860 0x7fdf8419a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:41.262 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:34.166+0000 7fdf82ffd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdf84104860 0x7fdf8419a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:46400/0 (socket says 172.21.15.19:46400) 2024-01-24T14:19:41.262 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:34.166+0000 7fdf82ffd700 1 -- 172.21.15.19:0/3016077589 learned_addr learned my addr 172.21.15.19:0/3016077589 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:19:41.262 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:34.166+0000 7fdf82ffd700 1 -- 172.21.15.19:0/3016077589 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf74009550 con 0x7fdf84104860 2024-01-24T14:19:41.262 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:34.166+0000 7fdf82ffd700 1 --2- 172.21.15.19:0/3016077589 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdf84104860 0x7fdf8419a130 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdf74006af0 tx=0x7fdf74011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:41.262 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:34.167+0000 7fdf6bfff700 1 -- 172.21.15.19:0/3016077589 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdf74003a40 con 0x7fdf84104860 2024-01-24T14:19:41.262 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:34.167+0000 7fdf6bfff700 1 -- 172.21.15.19:0/3016077589 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdf74003ba0 con 0x7fdf84104860 2024-01-24T14:19:41.262 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:34.167+0000 7fdf89561700 1 -- 172.21.15.19:0/3016077589 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf8419a870 con 0x7fdf84104860 2024-01-24T14:19:41.262 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:34.167+0000 7fdf6bfff700 1 -- 172.21.15.19:0/3016077589 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdf7401a700 con 0x7fdf84104860 2024-01-24T14:19:41.262 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:34.167+0000 7fdf89561700 1 -- 172.21.15.19:0/3016077589 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf8419ac90 con 0x7fdf84104860 2024-01-24T14:19:41.262 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:34.168+0000 7fdf6bfff700 1 -- 172.21.15.19:0/3016077589 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) v1 ==== 45268+0+0 (secure 0 0 0) 0x7fdf7401a980 con 0x7fdf84104860 2024-01-24T14:19:41.263 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:34.168+0000 7fdf6bfff700 1 --2- 172.21.15.19:0/3016077589 >> [v2:172.21.15.19:6800/2754937713,v1:172.21.15.19:6801/2754937713] conn(0x7fdf6c0383c0 0x7fdf6c03a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:41.263 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:34.168+0000 7fdf6bfff700 1 -- 172.21.15.19:0/3016077589 --> [v2:172.21.15.19:6800/2754937713,v1:172.21.15.19:6801/2754937713] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdf6c03af90 con 0x7fdf6c0383c0 2024-01-24T14:19:41.263 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:34.168+0000 7fdf6bfff700 1 -- 172.21.15.19:0/3016077589 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fdf7404c130 con 0x7fdf84104860 2024-01-24T14:19:41.263 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:34.168+0000 7fdf827fc700 1 -- 172.21.15.19:0/3016077589 >> [v2:172.21.15.19:6800/2754937713,v1:172.21.15.19:6801/2754937713] conn(0x7fdf6c0383c0 msgr2=0x7fdf6c03a880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:6800/2754937713 2024-01-24T14:19:41.263 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:34.168+0000 7fdf827fc700 1 --2- 172.21.15.19:0/3016077589 >> [v2:172.21.15.19:6800/2754937713,v1:172.21.15.19:6801/2754937713] conn(0x7fdf6c0383c0 0x7fdf6c03a880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-24T14:19:41.263 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:34.369+0000 7fdf827fc700 1 -- 172.21.15.19:0/3016077589 >> [v2:172.21.15.19:6800/2754937713,v1:172.21.15.19:6801/2754937713] conn(0x7fdf6c0383c0 msgr2=0x7fdf6c03a880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:6800/2754937713 2024-01-24T14:19:41.263 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:34.369+0000 7fdf827fc700 1 --2- 172.21.15.19:0/3016077589 >> [v2:172.21.15.19:6800/2754937713,v1:172.21.15.19:6801/2754937713] conn(0x7fdf6c0383c0 0x7fdf6c03a880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-24T14:19:41.263 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:34.770+0000 7fdf827fc700 1 -- 172.21.15.19:0/3016077589 >> [v2:172.21.15.19:6800/2754937713,v1:172.21.15.19:6801/2754937713] conn(0x7fdf6c0383c0 msgr2=0x7fdf6c03a880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:6800/2754937713 2024-01-24T14:19:41.263 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:34.770+0000 7fdf827fc700 1 --2- 172.21.15.19:0/3016077589 >> [v2:172.21.15.19:6800/2754937713,v1:172.21.15.19:6801/2754937713] conn(0x7fdf6c0383c0 0x7fdf6c03a880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-24T14:19:41.263 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:35.571+0000 7fdf827fc700 1 -- 172.21.15.19:0/3016077589 >> [v2:172.21.15.19:6800/2754937713,v1:172.21.15.19:6801/2754937713] conn(0x7fdf6c0383c0 msgr2=0x7fdf6c03a880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:6800/2754937713 2024-01-24T14:19:41.263 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:35.571+0000 7fdf827fc700 1 --2- 172.21.15.19:0/3016077589 >> [v2:172.21.15.19:6800/2754937713,v1:172.21.15.19:6801/2754937713] conn(0x7fdf6c0383c0 0x7fdf6c03a880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-24T14:19:41.263 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:37.172+0000 7fdf827fc700 1 -- 172.21.15.19:0/3016077589 >> [v2:172.21.15.19:6800/2754937713,v1:172.21.15.19:6801/2754937713] conn(0x7fdf6c0383c0 msgr2=0x7fdf6c03a880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:6800/2754937713 2024-01-24T14:19:41.263 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:37.172+0000 7fdf827fc700 1 --2- 172.21.15.19:0/3016077589 >> [v2:172.21.15.19:6800/2754937713,v1:172.21.15.19:6801/2754937713] conn(0x7fdf6c0383c0 0x7fdf6c03a880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-01-24T14:19:41.263 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:39.854+0000 7fdf6bfff700 1 -- 172.21.15.19:0/3016077589 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mgrmap(e 6) v1 ==== 45035+0+0 (secure 0 0 0) 0x7fdf7402d960 con 0x7fdf84104860 2024-01-24T14:19:41.264 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:39.854+0000 7fdf6bfff700 1 -- 172.21.15.19:0/3016077589 >> [v2:172.21.15.19:6800/2754937713,v1:172.21.15.19:6801/2754937713] conn(0x7fdf6c0383c0 msgr2=0x7fdf6c03a880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-24T14:19:41.264 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:39.854+0000 7fdf6bfff700 1 --2- 172.21.15.19:0/3016077589 >> [v2:172.21.15.19:6800/2754937713,v1:172.21.15.19:6801/2754937713] conn(0x7fdf6c0383c0 0x7fdf6c03a880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:41.264 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:40.855+0000 7fdf6bfff700 1 -- 172.21.15.19:0/3016077589 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mgrmap(e 7) v1 ==== 45162+0+0 (secure 0 0 0) 0x7fdf7404dc40 con 0x7fdf84104860 2024-01-24T14:19:41.264 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:40.855+0000 7fdf6bfff700 1 --2- 172.21.15.19:0/3016077589 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7fdf6c03bca0 0x7fdf6c03e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:41.264 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:40.855+0000 7fdf6bfff700 1 -- 172.21.15.19:0/3016077589 --> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdf6c03af90 con 0x7fdf6c03bca0 2024-01-24T14:19:41.264 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:40.856+0000 7fdf827fc700 1 --2- 172.21.15.19:0/3016077589 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7fdf6c03bca0 0x7fdf6c03e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:41.264 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:40.857+0000 7fdf827fc700 1 --2- 172.21.15.19:0/3016077589 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7fdf6c03bca0 0x7fdf6c03e090 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fdf78003a10 tx=0x7fdf78009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:41.264 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:40.857+0000 7fdf6bfff700 1 -- 172.21.15.19:0/3016077589 <== mgr.14120 v2:172.21.15.19:6800/3758061162 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7fdf6c03af90 con 0x7fdf6c03bca0 2024-01-24T14:19:41.264 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:40.864+0000 7fdf89561700 1 -- 172.21.15.19:0/3016077589 --> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fdf8419b470 con 0x7fdf6c03bca0 2024-01-24T14:19:41.264 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:40.864+0000 7fdf6bfff700 1 -- 172.21.15.19:0/3016077589 <== mgr.14120 v2:172.21.15.19:6800/3758061162 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fdf8419b470 con 0x7fdf6c03bca0 2024-01-24T14:19:41.264 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:40.864+0000 7fdf89561700 1 -- 172.21.15.19:0/3016077589 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7fdf6c03bca0 msgr2=0x7fdf6c03e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:41.264 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:40.864+0000 7fdf89561700 1 --2- 172.21.15.19:0/3016077589 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7fdf6c03bca0 0x7fdf6c03e090 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fdf78003a10 tx=0x7fdf78009290 comp rx=0 tx=0).stop 2024-01-24T14:19:41.264 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:40.864+0000 7fdf89561700 1 -- 172.21.15.19:0/3016077589 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdf84104860 msgr2=0x7fdf8419a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:41.264 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:40.864+0000 7fdf89561700 1 --2- 172.21.15.19:0/3016077589 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdf84104860 0x7fdf8419a130 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdf74006af0 tx=0x7fdf74011ad0 comp rx=0 tx=0).stop 2024-01-24T14:19:41.264 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:40.864+0000 7fdf89561700 1 -- 172.21.15.19:0/3016077589 shutdown_connections 2024-01-24T14:19:41.265 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:40.864+0000 7fdf89561700 1 --2- 172.21.15.19:0/3016077589 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7fdf6c03bca0 0x7fdf6c03e090 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:41.265 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:40.864+0000 7fdf89561700 1 --2- 172.21.15.19:0/3016077589 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdf84104860 0x7fdf8419a130 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:41.265 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:40.864+0000 7fdf89561700 1 --2- 172.21.15.19:0/3016077589 >> [v2:172.21.15.19:6800/2754937713,v1:172.21.15.19:6801/2754937713] conn(0x7fdf6c0383c0 0x7fdf6c03a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:41.265 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:40.864+0000 7fdf89561700 1 -- 172.21.15.19:0/3016077589 >> 172.21.15.19:0/3016077589 conn(0x7fdf84100190 msgr2=0x7fdf84100ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:19:41.265 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:40.865+0000 7fdf89561700 1 -- 172.21.15.19:0/3016077589 shutdown_connections 2024-01-24T14:19:41.265 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:40.865+0000 7fdf89561700 1 -- 172.21.15.19:0/3016077589 wait complete. 2024-01-24T14:19:41.265 INFO:teuthology.orchestra.run.smithi019.stdout:mgr epoch 5 is available 2024-01-24T14:19:41.265 INFO:teuthology.orchestra.run.smithi019.stdout:Setting orchestrator backend to cephadm... 2024-01-24T14:19:41.406 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:41 smithi019 ceph-mon[40663]: Found migration_current of "None". Setting to last migration. 2024-01-24T14:19:41.406 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:41 smithi019 ceph-mon[40663]: from='mgr.14120 172.21.15.19:0/1799968416' entity='mgr.x' 2024-01-24T14:19:41.407 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:41 smithi019 ceph-mon[40663]: from='mgr.14120 172.21.15.19:0/1799968416' entity='mgr.x' 2024-01-24T14:19:41.407 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:41 smithi019 ceph-mon[40663]: mgrmap e7: x(active, since 1.00595s) 2024-01-24T14:19:42.243 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.645+0000 7fbc68ced700 1 Processor -- start 2024-01-24T14:19:42.243 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.645+0000 7fbc68ced700 1 -- start start 2024-01-24T14:19:42.243 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.646+0000 7fbc68ced700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbc64107030 0x7fbc64109420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:42.243 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.646+0000 7fbc68ced700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc64079e70 con 0x7fbc64107030 2024-01-24T14:19:42.243 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.646+0000 7fbc6259c700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbc64107030 0x7fbc64109420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:42.243 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.646+0000 7fbc6259c700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbc64107030 0x7fbc64109420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:55584/0 (socket says 172.21.15.19:55584) 2024-01-24T14:19:42.243 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.646+0000 7fbc6259c700 1 -- 172.21.15.19:0/92780343 learned_addr learned my addr 172.21.15.19:0/92780343 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:19:42.243 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.646+0000 7fbc6259c700 1 -- 172.21.15.19:0/92780343 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc64109960 con 0x7fbc64107030 2024-01-24T14:19:42.243 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.646+0000 7fbc6259c700 1 --2- 172.21.15.19:0/92780343 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbc64107030 0x7fbc64109420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbc540098a0 tx=0x7fbc54009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc660085a7c3872d server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:42.243 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.647+0000 7fbc6159a700 1 -- 172.21.15.19:0/92780343 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbc54004160 con 0x7fbc64107030 2024-01-24T14:19:42.244 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.647+0000 7fbc6159a700 1 -- 172.21.15.19:0/92780343 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbc540042c0 con 0x7fbc64107030 2024-01-24T14:19:42.244 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.647+0000 7fbc6159a700 1 -- 172.21.15.19:0/92780343 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbc54011a00 con 0x7fbc64107030 2024-01-24T14:19:42.244 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.647+0000 7fbc68ced700 1 -- 172.21.15.19:0/92780343 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbc64107030 msgr2=0x7fbc64109420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:42.244 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.647+0000 7fbc68ced700 1 --2- 172.21.15.19:0/92780343 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbc64107030 0x7fbc64109420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbc540098a0 tx=0x7fbc54009bb0 comp rx=0 tx=0).stop 2024-01-24T14:19:42.244 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.647+0000 7fbc68ced700 1 -- 172.21.15.19:0/92780343 shutdown_connections 2024-01-24T14:19:42.244 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.647+0000 7fbc68ced700 1 --2- 172.21.15.19:0/92780343 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbc64107030 0x7fbc64109420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:42.244 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.647+0000 7fbc68ced700 1 -- 172.21.15.19:0/92780343 >> 172.21.15.19:0/92780343 conn(0x7fbc64100a40 msgr2=0x7fbc64102e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:19:42.244 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.647+0000 7fbc68ced700 1 -- 172.21.15.19:0/92780343 shutdown_connections 2024-01-24T14:19:42.244 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.647+0000 7fbc68ced700 1 -- 172.21.15.19:0/92780343 wait complete. 2024-01-24T14:19:42.244 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.648+0000 7fbc68ced700 1 Processor -- start 2024-01-24T14:19:42.244 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.649+0000 7fbc68ced700 1 -- start start 2024-01-24T14:19:42.244 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.649+0000 7fbc68ced700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbc64107030 0x7fbc64106990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:42.245 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.649+0000 7fbc68ced700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc64106ed0 con 0x7fbc64107030 2024-01-24T14:19:42.245 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.649+0000 7fbc6259c700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbc64107030 0x7fbc64106990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:42.245 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.649+0000 7fbc6259c700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbc64107030 0x7fbc64106990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:55596/0 (socket says 172.21.15.19:55596) 2024-01-24T14:19:42.245 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.649+0000 7fbc6259c700 1 -- 172.21.15.19:0/3027581459 learned_addr learned my addr 172.21.15.19:0/3027581459 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:19:42.245 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.649+0000 7fbc6259c700 1 -- 172.21.15.19:0/3027581459 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc54009550 con 0x7fbc64107030 2024-01-24T14:19:42.245 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.649+0000 7fbc6259c700 1 --2- 172.21.15.19:0/3027581459 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbc64107030 0x7fbc64106990 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbc54009fb0 tx=0x7fbc54011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:42.245 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.650+0000 7fbc537fe700 1 -- 172.21.15.19:0/3027581459 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbc54003a40 con 0x7fbc64107030 2024-01-24T14:19:42.245 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.650+0000 7fbc537fe700 1 -- 172.21.15.19:0/3027581459 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbc54003ba0 con 0x7fbc64107030 2024-01-24T14:19:42.245 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.650+0000 7fbc68ced700 1 -- 172.21.15.19:0/3027581459 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc64104fe0 con 0x7fbc64107030 2024-01-24T14:19:42.245 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.650+0000 7fbc537fe700 1 -- 172.21.15.19:0/3027581459 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbc5401a750 con 0x7fbc64107030 2024-01-24T14:19:42.245 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.650+0000 7fbc68ced700 1 -- 172.21.15.19:0/3027581459 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc64105400 con 0x7fbc64107030 2024-01-24T14:19:42.245 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.650+0000 7fbc537fe700 1 -- 172.21.15.19:0/3027581459 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 7) v1 ==== 45162+0+0 (secure 0 0 0) 0x7fbc5401a9d0 con 0x7fbc64107030 2024-01-24T14:19:42.245 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.651+0000 7fbc537fe700 1 --2- 172.21.15.19:0/3027581459 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7fbc4c038250 0x7fbc4c03a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:42.246 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.651+0000 7fbc537fe700 1 -- 172.21.15.19:0/3027581459 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fbc5404c240 con 0x7fbc64107030 2024-01-24T14:19:42.246 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.651+0000 7fbc68ced700 1 -- 172.21.15.19:0/3027581459 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc6410a700 con 0x7fbc64107030 2024-01-24T14:19:42.246 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.651+0000 7fbc61d9b700 1 --2- 172.21.15.19:0/3027581459 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7fbc4c038250 0x7fbc4c03a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:42.246 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.651+0000 7fbc61d9b700 1 --2- 172.21.15.19:0/3027581459 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7fbc4c038250 0x7fbc4c03a710 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fbc58006fd0 tx=0x7fbc58006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:42.246 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.656+0000 7fbc537fe700 1 -- 172.21.15.19:0/3027581459 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fbc54023390 con 0x7fbc64107030 2024-01-24T14:19:42.246 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.879+0000 7fbc68ced700 1 -- 172.21.15.19:0/3027581459 --> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fbc64101ea0 con 0x7fbc4c038250 2024-01-24T14:19:42.246 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.885+0000 7fbc537fe700 1 -- 172.21.15.19:0/3027581459 <== mgr.14120 v2:172.21.15.19:6800/3758061162 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fbc64101ea0 con 0x7fbc4c038250 2024-01-24T14:19:42.246 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.888+0000 7fbc68ced700 1 -- 172.21.15.19:0/3027581459 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7fbc4c038250 msgr2=0x7fbc4c03a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:42.246 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.888+0000 7fbc68ced700 1 --2- 172.21.15.19:0/3027581459 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7fbc4c038250 0x7fbc4c03a710 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fbc58006fd0 tx=0x7fbc58006e40 comp rx=0 tx=0).stop 2024-01-24T14:19:42.246 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.888+0000 7fbc68ced700 1 -- 172.21.15.19:0/3027581459 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbc64107030 msgr2=0x7fbc64106990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:42.246 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.888+0000 7fbc68ced700 1 --2- 172.21.15.19:0/3027581459 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbc64107030 0x7fbc64106990 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbc54009fb0 tx=0x7fbc54011ad0 comp rx=0 tx=0).stop 2024-01-24T14:19:42.247 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.888+0000 7fbc68ced700 1 -- 172.21.15.19:0/3027581459 shutdown_connections 2024-01-24T14:19:42.247 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.888+0000 7fbc68ced700 1 --2- 172.21.15.19:0/3027581459 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7fbc4c038250 0x7fbc4c03a710 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:42.247 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.888+0000 7fbc68ced700 1 --2- 172.21.15.19:0/3027581459 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbc64107030 0x7fbc64106990 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:42.247 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.888+0000 7fbc68ced700 1 -- 172.21.15.19:0/3027581459 >> 172.21.15.19:0/3027581459 conn(0x7fbc64100a40 msgr2=0x7fbc64101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:19:42.247 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.888+0000 7fbc68ced700 1 -- 172.21.15.19:0/3027581459 shutdown_connections 2024-01-24T14:19:42.247 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:41.888+0000 7fbc68ced700 1 -- 172.21.15.19:0/3027581459 wait complete. 2024-01-24T14:19:42.485 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:42 smithi019 ceph-mon[40663]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-01-24T14:19:42.485 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:42 smithi019 ceph-mon[40663]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-01-24T14:19:42.485 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:42 smithi019 ceph-mon[40663]: from='mgr.14120 172.21.15.19:0/1799968416' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:19:42.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:42 smithi019 ceph-mon[40663]: from='mgr.14120 172.21.15.19:0/1799968416' entity='mgr.x' 2024-01-24T14:19:42.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:42 smithi019 ceph-mon[40663]: from='mgr.14120 172.21.15.19:0/1799968416' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:19:43.192 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout value unchanged 2024-01-24T14:19:43.192 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.610+0000 7f47c2c35700 1 Processor -- start 2024-01-24T14:19:43.192 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.610+0000 7f47c2c35700 1 -- start start 2024-01-24T14:19:43.192 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.610+0000 7f47c2c35700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f47bc107040 0x7f47bc109430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:43.192 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.610+0000 7f47c2c35700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47bc079e70 con 0x7f47bc107040 2024-01-24T14:19:43.192 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.610+0000 7f47c09d1700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f47bc107040 0x7f47bc109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:43.193 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.610+0000 7f47c09d1700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f47bc107040 0x7f47bc109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:55610/0 (socket says 172.21.15.19:55610) 2024-01-24T14:19:43.193 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.610+0000 7f47c09d1700 1 -- 172.21.15.19:0/3681648309 learned_addr learned my addr 172.21.15.19:0/3681648309 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:19:43.193 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.611+0000 7f47c09d1700 1 -- 172.21.15.19:0/3681648309 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47bc109970 con 0x7f47bc107040 2024-01-24T14:19:43.193 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.611+0000 7f47c09d1700 1 --2- 172.21.15.19:0/3681648309 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f47bc107040 0x7f47bc109430 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f47b80098a0 tx=0x7f47b8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef19637f1b44871a server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:43.193 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.611+0000 7f47b37fe700 1 -- 172.21.15.19:0/3681648309 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f47b8004160 con 0x7f47bc107040 2024-01-24T14:19:43.193 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.611+0000 7f47b37fe700 1 -- 172.21.15.19:0/3681648309 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f47b80042c0 con 0x7f47bc107040 2024-01-24T14:19:43.193 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.611+0000 7f47b37fe700 1 -- 172.21.15.19:0/3681648309 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f47b8011a00 con 0x7f47bc107040 2024-01-24T14:19:43.193 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.611+0000 7f47c2c35700 1 -- 172.21.15.19:0/3681648309 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f47bc107040 msgr2=0x7f47bc109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:43.193 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.611+0000 7f47c2c35700 1 --2- 172.21.15.19:0/3681648309 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f47bc107040 0x7f47bc109430 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f47b80098a0 tx=0x7f47b8009bb0 comp rx=0 tx=0).stop 2024-01-24T14:19:43.193 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.612+0000 7f47c2c35700 1 -- 172.21.15.19:0/3681648309 shutdown_connections 2024-01-24T14:19:43.193 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.612+0000 7f47c2c35700 1 --2- 172.21.15.19:0/3681648309 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f47bc107040 0x7f47bc109430 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:43.193 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.612+0000 7f47c2c35700 1 -- 172.21.15.19:0/3681648309 >> 172.21.15.19:0/3681648309 conn(0x7f47bc100a70 msgr2=0x7f47bc102e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:19:43.193 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.612+0000 7f47c2c35700 1 -- 172.21.15.19:0/3681648309 shutdown_connections 2024-01-24T14:19:43.193 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.612+0000 7f47c2c35700 1 -- 172.21.15.19:0/3681648309 wait complete. 2024-01-24T14:19:43.193 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.612+0000 7f47c2c35700 1 Processor -- start 2024-01-24T14:19:43.194 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.612+0000 7f47c2c35700 1 -- start start 2024-01-24T14:19:43.194 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.612+0000 7f47c2c35700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f47bc107040 0x7f47bc1069a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:43.194 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.612+0000 7f47c2c35700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47bc106ee0 con 0x7f47bc107040 2024-01-24T14:19:43.194 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.613+0000 7f47c09d1700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f47bc107040 0x7f47bc1069a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:43.194 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.613+0000 7f47c09d1700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f47bc107040 0x7f47bc1069a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:55612/0 (socket says 172.21.15.19:55612) 2024-01-24T14:19:43.194 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.613+0000 7f47c09d1700 1 -- 172.21.15.19:0/2522629782 learned_addr learned my addr 172.21.15.19:0/2522629782 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:19:43.194 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.613+0000 7f47c09d1700 1 -- 172.21.15.19:0/2522629782 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b8009550 con 0x7f47bc107040 2024-01-24T14:19:43.194 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.613+0000 7f47c09d1700 1 --2- 172.21.15.19:0/2522629782 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f47bc107040 0x7f47bc1069a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f47b8009130 tx=0x7f47b8011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:43.194 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.613+0000 7f47b1ffb700 1 -- 172.21.15.19:0/2522629782 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f47b8003b70 con 0x7f47bc107040 2024-01-24T14:19:43.194 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.613+0000 7f47b1ffb700 1 -- 172.21.15.19:0/2522629782 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f47b8003cd0 con 0x7f47bc107040 2024-01-24T14:19:43.194 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.613+0000 7f47c2c35700 1 -- 172.21.15.19:0/2522629782 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47bc104ff0 con 0x7f47bc107040 2024-01-24T14:19:43.194 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.613+0000 7f47b1ffb700 1 -- 172.21.15.19:0/2522629782 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f47b80197d0 con 0x7f47bc107040 2024-01-24T14:19:43.194 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.614+0000 7f47c2c35700 1 -- 172.21.15.19:0/2522629782 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47bc105410 con 0x7f47bc107040 2024-01-24T14:19:43.194 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.614+0000 7f47b1ffb700 1 -- 172.21.15.19:0/2522629782 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 8) v1 ==== 45268+0+0 (secure 0 0 0) 0x7f47b801a430 con 0x7f47bc107040 2024-01-24T14:19:43.195 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.615+0000 7f47b1ffb700 1 --2- 172.21.15.19:0/2522629782 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7f47a4038460 0x7f47a403a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:43.195 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.615+0000 7f47c2c35700 1 -- 172.21.15.19:0/2522629782 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47bc10a710 con 0x7f47bc107040 2024-01-24T14:19:43.195 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.615+0000 7f47b1ffb700 1 -- 172.21.15.19:0/2522629782 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f47b804c470 con 0x7f47bc107040 2024-01-24T14:19:43.195 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.615+0000 7f47b3fff700 1 --2- 172.21.15.19:0/2522629782 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7f47a4038460 0x7f47a403a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:43.195 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.615+0000 7f47b3fff700 1 --2- 172.21.15.19:0/2522629782 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7f47a4038460 0x7f47a403a920 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f47ac006fd0 tx=0x7f47ac006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:43.195 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.620+0000 7f47b1ffb700 1 -- 172.21.15.19:0/2522629782 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f47b807c120 con 0x7f47bc107040 2024-01-24T14:19:43.195 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.828+0000 7f47c2c35700 1 -- 172.21.15.19:0/2522629782 --> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f47bc101ee0 con 0x7f47a4038460 2024-01-24T14:19:43.195 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.829+0000 7f47b1ffb700 1 -- 172.21.15.19:0/2522629782 <== mgr.14120 v2:172.21.15.19:6800/3758061162 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f47bc101ee0 con 0x7f47a4038460 2024-01-24T14:19:43.195 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.831+0000 7f47c2c35700 1 -- 172.21.15.19:0/2522629782 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7f47a4038460 msgr2=0x7f47a403a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:43.195 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.831+0000 7f47c2c35700 1 --2- 172.21.15.19:0/2522629782 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7f47a4038460 0x7f47a403a920 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f47ac006fd0 tx=0x7f47ac006e40 comp rx=0 tx=0).stop 2024-01-24T14:19:43.195 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.831+0000 7f47c2c35700 1 -- 172.21.15.19:0/2522629782 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f47bc107040 msgr2=0x7f47bc1069a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:43.195 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.831+0000 7f47c2c35700 1 --2- 172.21.15.19:0/2522629782 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f47bc107040 0x7f47bc1069a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f47b8009130 tx=0x7f47b8011ad0 comp rx=0 tx=0).stop 2024-01-24T14:19:43.195 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.831+0000 7f47c2c35700 1 -- 172.21.15.19:0/2522629782 shutdown_connections 2024-01-24T14:19:43.195 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.831+0000 7f47c2c35700 1 --2- 172.21.15.19:0/2522629782 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7f47a4038460 0x7f47a403a920 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:43.196 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.831+0000 7f47c2c35700 1 --2- 172.21.15.19:0/2522629782 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f47bc107040 0x7f47bc1069a0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:43.196 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.831+0000 7f47c2c35700 1 -- 172.21.15.19:0/2522629782 >> 172.21.15.19:0/2522629782 conn(0x7f47bc100a70 msgr2=0x7f47bc1017d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:19:43.196 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.831+0000 7f47c2c35700 1 -- 172.21.15.19:0/2522629782 shutdown_connections 2024-01-24T14:19:43.196 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:42.831+0000 7f47c2c35700 1 -- 172.21.15.19:0/2522629782 wait complete. 2024-01-24T14:19:43.196 INFO:teuthology.orchestra.run.smithi019.stdout:Generating ssh key... 2024-01-24T14:19:43.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:43 smithi019 ceph-mon[40663]: [24/Jan/2024:14:19:41] ENGINE Bus STARTING 2024-01-24T14:19:43.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:43 smithi019 ceph-mon[40663]: [24/Jan/2024:14:19:41] ENGINE Serving on https://172.21.15.19:7150 2024-01-24T14:19:43.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:43 smithi019 ceph-mon[40663]: [24/Jan/2024:14:19:41] ENGINE Serving on http://172.21.15.19:8765 2024-01-24T14:19:43.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:43 smithi019 ceph-mon[40663]: [24/Jan/2024:14:19:41] ENGINE Bus STARTED 2024-01-24T14:19:43.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:43 smithi019 ceph-mon[40663]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:19:43.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:43 smithi019 ceph-mon[40663]: mgrmap e8: x(active, since 2s) 2024-01-24T14:19:44.323 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:44 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: Generating public/private rsa key pair. 2024-01-24T14:19:44.323 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:44 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: Your identification has been saved in /tmp/tmp43n2wroy/key. 2024-01-24T14:19:44.323 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:44 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: Your public key has been saved in /tmp/tmp43n2wroy/key.pub. 2024-01-24T14:19:44.323 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:44 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: The key fingerprint is: 2024-01-24T14:19:44.323 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:44 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: SHA256:j9zGEZQ0Em9eEUVIIaNs7jJ10aVpqR5170o4+k8B5K0 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715 2024-01-24T14:19:44.323 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:44 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: The key's randomart image is: 2024-01-24T14:19:44.323 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:44 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: +---[RSA 3072]----+ 2024-01-24T14:19:44.323 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:44 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: | oo*o**+ | 2024-01-24T14:19:44.323 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:44 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: | . =.B.B | 2024-01-24T14:19:44.323 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:44 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: | + = @ o | 2024-01-24T14:19:44.323 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:44 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: | o o B + . | 2024-01-24T14:19:44.323 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:44 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: | S * E . .| 2024-01-24T14:19:44.324 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:44 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: | + B o . o | 2024-01-24T14:19:44.324 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:44 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: | o + * o o .| 2024-01-24T14:19:44.324 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:44 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: | o . . + . | 2024-01-24T14:19:44.324 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:44 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: | ....o | 2024-01-24T14:19:44.324 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:44 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: +----[SHA256]-----+ 2024-01-24T14:19:44.408 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:43.579+0000 7efdd301e700 1 Processor -- start 2024-01-24T14:19:44.408 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:43.579+0000 7efdd301e700 1 -- start start 2024-01-24T14:19:44.408 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:43.580+0000 7efdd301e700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efdcc104e70 0x7efdcc105290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:44.408 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:43.580+0000 7efdd301e700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdcc079e30 con 0x7efdcc104e70 2024-01-24T14:19:44.408 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:43.580+0000 7efdd0dba700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efdcc104e70 0x7efdcc105290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:44.408 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:43.580+0000 7efdd0dba700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efdcc104e70 0x7efdcc105290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:55616/0 (socket says 172.21.15.19:55616) 2024-01-24T14:19:44.408 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:43.580+0000 7efdd0dba700 1 -- 172.21.15.19:0/4181850401 learned_addr learned my addr 172.21.15.19:0/4181850401 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:19:44.408 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:43.580+0000 7efdd0dba700 1 -- 172.21.15.19:0/4181850401 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdcc1057d0 con 0x7efdcc104e70 2024-01-24T14:19:44.409 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:43.580+0000 7efdd0dba700 1 --2- 172.21.15.19:0/4181850401 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efdcc104e70 0x7efdcc105290 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7efdbc0098a0 tx=0x7efdbc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaf2c860a6bb1087 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:44.409 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:43.581+0000 7efdcb7fe700 1 -- 172.21.15.19:0/4181850401 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efdbc004160 con 0x7efdcc104e70 2024-01-24T14:19:44.409 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:43.581+0000 7efdcb7fe700 1 -- 172.21.15.19:0/4181850401 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efdbc0042c0 con 0x7efdcc104e70 2024-01-24T14:19:44.409 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:43.581+0000 7efdcb7fe700 1 -- 172.21.15.19:0/4181850401 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efdbc011a00 con 0x7efdcc104e70 2024-01-24T14:19:44.409 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:43.581+0000 7efdd301e700 1 -- 172.21.15.19:0/4181850401 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efdcc104e70 msgr2=0x7efdcc105290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:44.409 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:43.581+0000 7efdd301e700 1 --2- 172.21.15.19:0/4181850401 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efdcc104e70 0x7efdcc105290 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7efdbc0098a0 tx=0x7efdbc009bb0 comp rx=0 tx=0).stop 2024-01-24T14:19:44.409 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:43.581+0000 7efdd301e700 1 -- 172.21.15.19:0/4181850401 shutdown_connections 2024-01-24T14:19:44.409 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:43.581+0000 7efdd301e700 1 --2- 172.21.15.19:0/4181850401 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efdcc104e70 0x7efdcc105290 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:44.409 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:43.581+0000 7efdd301e700 1 -- 172.21.15.19:0/4181850401 >> 172.21.15.19:0/4181850401 conn(0x7efdcc100a70 msgr2=0x7efdcc102e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:19:44.409 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:43.581+0000 7efdd301e700 1 -- 172.21.15.19:0/4181850401 shutdown_connections 2024-01-24T14:19:44.410 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:43.581+0000 7efdd301e700 1 -- 172.21.15.19:0/4181850401 wait complete. 2024-01-24T14:19:44.410 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:43.582+0000 7efdd301e700 1 Processor -- start 2024-01-24T14:19:44.410 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:43.582+0000 7efdd301e700 1 -- start start 2024-01-24T14:19:44.410 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:43.582+0000 7efdd301e700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efdcc104e70 0x7efdcc19c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:44.410 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:43.582+0000 7efdd301e700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdcc19c900 con 0x7efdcc104e70 2024-01-24T14:19:44.410 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:43.582+0000 7efdd0dba700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efdcc104e70 0x7efdcc19c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:44.410 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:43.582+0000 7efdd0dba700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efdcc104e70 0x7efdcc19c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:55632/0 (socket says 172.21.15.19:55632) 2024-01-24T14:19:44.410 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:43.582+0000 7efdd0dba700 1 -- 172.21.15.19:0/2330882992 learned_addr learned my addr 172.21.15.19:0/2330882992 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:19:44.410 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:43.583+0000 7efdd0dba700 1 -- 172.21.15.19:0/2330882992 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdbc009550 con 0x7efdcc104e70 2024-01-24T14:19:44.410 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:43.583+0000 7efdd0dba700 1 --2- 172.21.15.19:0/2330882992 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efdcc104e70 0x7efdcc19c3c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7efdbc009130 tx=0x7efdbc011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:44.410 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:43.583+0000 7efdc9ffb700 1 -- 172.21.15.19:0/2330882992 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efdbc003b70 con 0x7efdcc104e70 2024-01-24T14:19:44.410 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:43.583+0000 7efdc9ffb700 1 -- 172.21.15.19:0/2330882992 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efdbc003cd0 con 0x7efdcc104e70 2024-01-24T14:19:44.410 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:43.583+0000 7efdd301e700 1 -- 172.21.15.19:0/2330882992 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdcc19cb00 con 0x7efdcc104e70 2024-01-24T14:19:44.410 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:43.583+0000 7efdc9ffb700 1 -- 172.21.15.19:0/2330882992 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efdbc0197d0 con 0x7efdcc104e70 2024-01-24T14:19:44.411 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:43.583+0000 7efdd301e700 1 -- 172.21.15.19:0/2330882992 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdcc19cf20 con 0x7efdcc104e70 2024-01-24T14:19:44.411 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:43.584+0000 7efdc9ffb700 1 -- 172.21.15.19:0/2330882992 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 8) v1 ==== 45268+0+0 (secure 0 0 0) 0x7efdbc0199c0 con 0x7efdcc104e70 2024-01-24T14:19:44.411 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:43.584+0000 7efdd301e700 1 -- 172.21.15.19:0/2330882992 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdcc195c50 con 0x7efdcc104e70 2024-01-24T14:19:44.411 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:43.584+0000 7efdc9ffb700 1 --2- 172.21.15.19:0/2330882992 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7efdb4038410 0x7efdb403a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:44.411 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:43.585+0000 7efdc9ffb700 1 -- 172.21.15.19:0/2330882992 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7efdbc04c5f0 con 0x7efdcc104e70 2024-01-24T14:19:44.411 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:43.585+0000 7efdcbfff700 1 --2- 172.21.15.19:0/2330882992 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7efdb4038410 0x7efdb403a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:44.411 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:43.585+0000 7efdcbfff700 1 --2- 172.21.15.19:0/2330882992 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7efdb4038410 0x7efdb403a8d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7efdc0006fd0 tx=0x7efdc0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:44.411 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:43.590+0000 7efdc9ffb700 1 -- 172.21.15.19:0/2330882992 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7efdbc018db0 con 0x7efdcc104e70 2024-01-24T14:19:44.411 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:43.800+0000 7efdd301e700 1 -- 172.21.15.19:0/2330882992 --> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7efdcc061180 con 0x7efdb4038410 2024-01-24T14:19:44.411 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.046+0000 7efdc9ffb700 1 -- 172.21.15.19:0/2330882992 <== mgr.14120 v2:172.21.15.19:6800/3758061162 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7efdcc061180 con 0x7efdb4038410 2024-01-24T14:19:44.411 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.048+0000 7efdd301e700 1 -- 172.21.15.19:0/2330882992 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7efdb4038410 msgr2=0x7efdb403a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:44.411 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.048+0000 7efdd301e700 1 --2- 172.21.15.19:0/2330882992 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7efdb4038410 0x7efdb403a8d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7efdc0006fd0 tx=0x7efdc0006e40 comp rx=0 tx=0).stop 2024-01-24T14:19:44.411 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.048+0000 7efdd301e700 1 -- 172.21.15.19:0/2330882992 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efdcc104e70 msgr2=0x7efdcc19c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:44.411 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.048+0000 7efdd301e700 1 --2- 172.21.15.19:0/2330882992 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efdcc104e70 0x7efdcc19c3c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7efdbc009130 tx=0x7efdbc011ad0 comp rx=0 tx=0).stop 2024-01-24T14:19:44.412 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.049+0000 7efdd301e700 1 -- 172.21.15.19:0/2330882992 shutdown_connections 2024-01-24T14:19:44.412 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.049+0000 7efdd301e700 1 --2- 172.21.15.19:0/2330882992 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7efdb4038410 0x7efdb403a8d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:44.412 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.049+0000 7efdd301e700 1 --2- 172.21.15.19:0/2330882992 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efdcc104e70 0x7efdcc19c3c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:44.412 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.049+0000 7efdd301e700 1 -- 172.21.15.19:0/2330882992 >> 172.21.15.19:0/2330882992 conn(0x7efdcc100a70 msgr2=0x7efdcc190a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:19:44.412 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.049+0000 7efdd301e700 1 -- 172.21.15.19:0/2330882992 shutdown_connections 2024-01-24T14:19:44.412 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.049+0000 7efdd301e700 1 -- 172.21.15.19:0/2330882992 wait complete. 2024-01-24T14:19:44.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:44 smithi019 ceph-mon[40663]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:19:44.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:44 smithi019 ceph-mon[40663]: from='mgr.14120 172.21.15.19:0/1799968416' entity='mgr.x' 2024-01-24T14:19:44.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:44 smithi019 ceph-mon[40663]: from='mgr.14120 172.21.15.19:0/1799968416' entity='mgr.x' 2024-01-24T14:19:45.341 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRBhvYBWmdATrSMvfWo6Pk/4oJflz89pkUJtpO7iZnUX31bWEnSrytuzacicjMV3MQ+QQkzGDZKaklWSetQGxNzyFpiRBVx+xlHz46XKSOd+8kYijwC/k0EGzuarszGkvIyvA/3EuKKs/QHQVcns1V3nPT+STZ6vjtJt0BlL5+x7VvYfnjyuwSgFLv7ipoMwCsV4IFbrxc1WrfKdNfjHf+lC3dzfI5gokxCrS+ZlCmzEP4vFrqqFKvIEvkDXsnZNuvZI4drAEIrhdUNYOmoiqOZZ1mH5WAv4K243R/lAFQbsSKkYXDAoEnYfDikIjFYalQUl7+VwbHIqNLi800cECQnsF6XO18CYeoMD85URhRmIxYnsHW4Yi24KVQq0ah/2knyYJMDJVkXwsJArFwMgD1kmVaSjaA+zgKIbfYnkOg1p0bPEcFLcDiZnkPEKM7CWIJDimupfbp4zXs/iZ0B0HOgj7UZPsLn0RWjfPly7tNaDjw1DTpFRXLs+AWQVEMGD8= ceph-654b0c8e-bac3-11ee-95b1-87774f69a715 2024-01-24T14:19:45.341 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.772+0000 7fb4db97c700 1 Processor -- start 2024-01-24T14:19:45.341 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.773+0000 7fb4db97c700 1 -- start start 2024-01-24T14:19:45.342 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.773+0000 7fb4db97c700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb4d4107050 0x7fb4d4109440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:45.342 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.773+0000 7fb4db97c700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d4079e50 con 0x7fb4d4107050 2024-01-24T14:19:45.342 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.773+0000 7fb4d9718700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb4d4107050 0x7fb4d4109440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:45.342 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.773+0000 7fb4d9718700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb4d4107050 0x7fb4d4109440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:55644/0 (socket says 172.21.15.19:55644) 2024-01-24T14:19:45.342 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.773+0000 7fb4d9718700 1 -- 172.21.15.19:0/2883110876 learned_addr learned my addr 172.21.15.19:0/2883110876 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:19:45.342 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.774+0000 7fb4d9718700 1 -- 172.21.15.19:0/2883110876 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4d4109980 con 0x7fb4d4107050 2024-01-24T14:19:45.342 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.774+0000 7fb4d9718700 1 --2- 172.21.15.19:0/2883110876 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb4d4107050 0x7fb4d4109440 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb4d00098a0 tx=0x7fb4d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0ce30433e4f92a0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:45.342 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.774+0000 7fb4cbfff700 1 -- 172.21.15.19:0/2883110876 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb4d0004160 con 0x7fb4d4107050 2024-01-24T14:19:45.342 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.774+0000 7fb4cbfff700 1 -- 172.21.15.19:0/2883110876 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb4d00042c0 con 0x7fb4d4107050 2024-01-24T14:19:45.342 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.774+0000 7fb4cbfff700 1 -- 172.21.15.19:0/2883110876 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb4d0011a00 con 0x7fb4d4107050 2024-01-24T14:19:45.342 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.775+0000 7fb4db97c700 1 -- 172.21.15.19:0/2883110876 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb4d4107050 msgr2=0x7fb4d4109440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:45.342 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.775+0000 7fb4db97c700 1 --2- 172.21.15.19:0/2883110876 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb4d4107050 0x7fb4d4109440 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb4d00098a0 tx=0x7fb4d0009bb0 comp rx=0 tx=0).stop 2024-01-24T14:19:45.342 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.775+0000 7fb4db97c700 1 -- 172.21.15.19:0/2883110876 shutdown_connections 2024-01-24T14:19:45.343 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.775+0000 7fb4db97c700 1 --2- 172.21.15.19:0/2883110876 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb4d4107050 0x7fb4d4109440 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:45.343 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.775+0000 7fb4db97c700 1 -- 172.21.15.19:0/2883110876 >> 172.21.15.19:0/2883110876 conn(0x7fb4d4100a40 msgr2=0x7fb4d4102e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:19:45.343 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.775+0000 7fb4db97c700 1 -- 172.21.15.19:0/2883110876 shutdown_connections 2024-01-24T14:19:45.343 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.775+0000 7fb4db97c700 1 -- 172.21.15.19:0/2883110876 wait complete. 2024-01-24T14:19:45.343 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.776+0000 7fb4db97c700 1 Processor -- start 2024-01-24T14:19:45.343 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.776+0000 7fb4db97c700 1 -- start start 2024-01-24T14:19:45.343 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.776+0000 7fb4db97c700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb4d4107050 0x7fb4d419a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:45.343 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.776+0000 7fb4db97c700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d419a710 con 0x7fb4d4107050 2024-01-24T14:19:45.343 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.776+0000 7fb4d9718700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb4d4107050 0x7fb4d419a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:45.343 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.776+0000 7fb4d9718700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb4d4107050 0x7fb4d419a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:55652/0 (socket says 172.21.15.19:55652) 2024-01-24T14:19:45.343 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.776+0000 7fb4d9718700 1 -- 172.21.15.19:0/3877303172 learned_addr learned my addr 172.21.15.19:0/3877303172 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:19:45.343 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.777+0000 7fb4d9718700 1 -- 172.21.15.19:0/3877303172 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4d0009550 con 0x7fb4d4107050 2024-01-24T14:19:45.343 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.777+0000 7fb4d9718700 1 --2- 172.21.15.19:0/3877303172 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb4d4107050 0x7fb4d419a1d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb4d0009fb0 tx=0x7fb4d0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:45.343 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.777+0000 7fb4ca7fc700 1 -- 172.21.15.19:0/3877303172 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb4d0003a40 con 0x7fb4d4107050 2024-01-24T14:19:45.344 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.777+0000 7fb4db97c700 1 -- 172.21.15.19:0/3877303172 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4d419a910 con 0x7fb4d4107050 2024-01-24T14:19:45.344 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.777+0000 7fb4ca7fc700 1 -- 172.21.15.19:0/3877303172 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb4d0003ba0 con 0x7fb4d4107050 2024-01-24T14:19:45.344 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.777+0000 7fb4ca7fc700 1 -- 172.21.15.19:0/3877303172 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb4d001a750 con 0x7fb4d4107050 2024-01-24T14:19:45.344 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.777+0000 7fb4db97c700 1 -- 172.21.15.19:0/3877303172 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4d419ad30 con 0x7fb4d4107050 2024-01-24T14:19:45.344 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.778+0000 7fb4ca7fc700 1 -- 172.21.15.19:0/3877303172 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 8) v1 ==== 45268+0+0 (secure 0 0 0) 0x7fb4d001a9d0 con 0x7fb4d4107050 2024-01-24T14:19:45.344 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.778+0000 7fb4db97c700 1 -- 172.21.15.19:0/3877303172 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4d4062370 con 0x7fb4d4107050 2024-01-24T14:19:45.344 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.778+0000 7fb4ca7fc700 1 --2- 172.21.15.19:0/3877303172 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7fb4c0038370 0x7fb4c003a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:45.344 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.778+0000 7fb4ca7fc700 1 -- 172.21.15.19:0/3877303172 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb4d004c210 con 0x7fb4d4107050 2024-01-24T14:19:45.344 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.778+0000 7fb4d8f17700 1 --2- 172.21.15.19:0/3877303172 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7fb4c0038370 0x7fb4c003a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:45.344 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.779+0000 7fb4d8f17700 1 --2- 172.21.15.19:0/3877303172 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7fb4c0038370 0x7fb4c003a830 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb4c4006fd0 tx=0x7fb4c4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:45.344 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.783+0000 7fb4ca7fc700 1 -- 172.21.15.19:0/3877303172 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fb4d0018d20 con 0x7fb4d4107050 2024-01-24T14:19:45.344 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.991+0000 7fb4db97c700 1 -- 172.21.15.19:0/3877303172 --> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fb4d4101eb0 con 0x7fb4c0038370 2024-01-24T14:19:45.344 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.991+0000 7fb4ca7fc700 1 -- 172.21.15.19:0/3877303172 <== mgr.14120 v2:172.21.15.19:6800/3758061162 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fb4d4101eb0 con 0x7fb4c0038370 2024-01-24T14:19:45.344 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.994+0000 7fb4db97c700 1 -- 172.21.15.19:0/3877303172 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7fb4c0038370 msgr2=0x7fb4c003a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:45.345 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.994+0000 7fb4db97c700 1 --2- 172.21.15.19:0/3877303172 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7fb4c0038370 0x7fb4c003a830 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb4c4006fd0 tx=0x7fb4c4006e40 comp rx=0 tx=0).stop 2024-01-24T14:19:45.345 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.994+0000 7fb4db97c700 1 -- 172.21.15.19:0/3877303172 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb4d4107050 msgr2=0x7fb4d419a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:45.345 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.994+0000 7fb4db97c700 1 --2- 172.21.15.19:0/3877303172 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb4d4107050 0x7fb4d419a1d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb4d0009fb0 tx=0x7fb4d0011ad0 comp rx=0 tx=0).stop 2024-01-24T14:19:45.345 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.994+0000 7fb4db97c700 1 -- 172.21.15.19:0/3877303172 shutdown_connections 2024-01-24T14:19:45.345 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.994+0000 7fb4db97c700 1 --2- 172.21.15.19:0/3877303172 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7fb4c0038370 0x7fb4c003a830 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:45.345 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.994+0000 7fb4db97c700 1 --2- 172.21.15.19:0/3877303172 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fb4d4107050 0x7fb4d419a1d0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:45.345 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.994+0000 7fb4db97c700 1 -- 172.21.15.19:0/3877303172 >> 172.21.15.19:0/3877303172 conn(0x7fb4d4100a40 msgr2=0x7fb4d41017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:19:45.345 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.994+0000 7fb4db97c700 1 -- 172.21.15.19:0/3877303172 shutdown_connections 2024-01-24T14:19:45.345 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:44.994+0000 7fb4db97c700 1 -- 172.21.15.19:0/3877303172 wait complete. 2024-01-24T14:19:45.345 INFO:teuthology.orchestra.run.smithi019.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-01-24T14:19:45.345 INFO:teuthology.orchestra.run.smithi019.stdout:Adding key to root@localhost authorized_keys... 2024-01-24T14:19:45.346 INFO:teuthology.orchestra.run.smithi019.stdout:Adding host smithi019... 2024-01-24T14:19:45.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:45 smithi019 ceph-mon[40663]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:19:45.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:45 smithi019 ceph-mon[40663]: Generating ssh key... 2024-01-24T14:19:46.485 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:46 smithi019 ceph-mon[40663]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:19:47.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:47 smithi019 ceph-mon[40663]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi019", "addr": "172.21.15.19", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:19:48.654 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:48 smithi019 ceph-mon[40663]: Deploying cephadm binary to smithi019 2024-01-24T14:19:48.700 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout Added host 'smithi019' with addr '172.21.15.19' 2024-01-24T14:19:48.700 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:45.700+0000 7ffa2fabc700 1 Processor -- start 2024-01-24T14:19:48.700 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:45.700+0000 7ffa2fabc700 1 -- start start 2024-01-24T14:19:48.700 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:45.701+0000 7ffa2fabc700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffa28107080 0x7ffa28109470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:48.701 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:45.701+0000 7ffa2fabc700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa28079e30 con 0x7ffa28107080 2024-01-24T14:19:48.701 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:45.701+0000 7ffa2d858700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffa28107080 0x7ffa28109470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:48.701 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:45.701+0000 7ffa2d858700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffa28107080 0x7ffa28109470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:55666/0 (socket says 172.21.15.19:55666) 2024-01-24T14:19:48.701 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:45.701+0000 7ffa2d858700 1 -- 172.21.15.19:0/2754925929 learned_addr learned my addr 172.21.15.19:0/2754925929 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:19:48.701 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:45.701+0000 7ffa2d858700 1 -- 172.21.15.19:0/2754925929 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa281099b0 con 0x7ffa28107080 2024-01-24T14:19:48.701 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:45.702+0000 7ffa2d858700 1 --2- 172.21.15.19:0/2754925929 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffa28107080 0x7ffa28109470 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ffa240098a0 tx=0x7ffa24009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd97046f14f02cfd server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:48.701 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:45.702+0000 7ffa2c856700 1 -- 172.21.15.19:0/2754925929 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffa24004160 con 0x7ffa28107080 2024-01-24T14:19:48.701 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:45.702+0000 7ffa2c856700 1 -- 172.21.15.19:0/2754925929 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffa240042c0 con 0x7ffa28107080 2024-01-24T14:19:48.701 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:45.702+0000 7ffa2c856700 1 -- 172.21.15.19:0/2754925929 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffa24011a00 con 0x7ffa28107080 2024-01-24T14:19:48.701 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:45.702+0000 7ffa2fabc700 1 -- 172.21.15.19:0/2754925929 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffa28107080 msgr2=0x7ffa28109470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:48.701 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:45.702+0000 7ffa2fabc700 1 --2- 172.21.15.19:0/2754925929 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffa28107080 0x7ffa28109470 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ffa240098a0 tx=0x7ffa24009bb0 comp rx=0 tx=0).stop 2024-01-24T14:19:48.701 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:45.702+0000 7ffa2fabc700 1 -- 172.21.15.19:0/2754925929 shutdown_connections 2024-01-24T14:19:48.701 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:45.703+0000 7ffa2fabc700 1 --2- 172.21.15.19:0/2754925929 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffa28107080 0x7ffa28109470 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:48.702 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:45.703+0000 7ffa2fabc700 1 -- 172.21.15.19:0/2754925929 >> 172.21.15.19:0/2754925929 conn(0x7ffa28100a70 msgr2=0x7ffa28102e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:19:48.702 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:45.703+0000 7ffa2fabc700 1 -- 172.21.15.19:0/2754925929 shutdown_connections 2024-01-24T14:19:48.702 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:45.703+0000 7ffa2fabc700 1 -- 172.21.15.19:0/2754925929 wait complete. 2024-01-24T14:19:48.702 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:45.703+0000 7ffa2fabc700 1 Processor -- start 2024-01-24T14:19:48.702 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:45.703+0000 7ffa2fabc700 1 -- start start 2024-01-24T14:19:48.702 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:45.703+0000 7ffa2fabc700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffa28107080 0x7ffa281069e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:48.702 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:45.703+0000 7ffa2fabc700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa28106f20 con 0x7ffa28107080 2024-01-24T14:19:48.702 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:45.704+0000 7ffa2d858700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffa28107080 0x7ffa281069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:48.702 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:45.704+0000 7ffa2d858700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffa28107080 0x7ffa281069e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:55670/0 (socket says 172.21.15.19:55670) 2024-01-24T14:19:48.702 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:45.704+0000 7ffa2d858700 1 -- 172.21.15.19:0/1919501821 learned_addr learned my addr 172.21.15.19:0/1919501821 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:19:48.702 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:45.704+0000 7ffa2d858700 1 -- 172.21.15.19:0/1919501821 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa24009550 con 0x7ffa28107080 2024-01-24T14:19:48.702 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:45.704+0000 7ffa2d858700 1 --2- 172.21.15.19:0/1919501821 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffa28107080 0x7ffa281069e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ffa24009130 tx=0x7ffa24011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:48.702 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:45.704+0000 7ffa1affd700 1 -- 172.21.15.19:0/1919501821 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffa24003b70 con 0x7ffa28107080 2024-01-24T14:19:48.702 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:45.704+0000 7ffa2fabc700 1 -- 172.21.15.19:0/1919501821 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa28105030 con 0x7ffa28107080 2024-01-24T14:19:48.703 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:45.704+0000 7ffa1affd700 1 -- 172.21.15.19:0/1919501821 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffa24003cd0 con 0x7ffa28107080 2024-01-24T14:19:48.703 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:45.704+0000 7ffa1affd700 1 -- 172.21.15.19:0/1919501821 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffa240197d0 con 0x7ffa28107080 2024-01-24T14:19:48.703 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:45.704+0000 7ffa2fabc700 1 -- 172.21.15.19:0/1919501821 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa28105450 con 0x7ffa28107080 2024-01-24T14:19:48.703 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:45.705+0000 7ffa1affd700 1 -- 172.21.15.19:0/1919501821 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 8) v1 ==== 45268+0+0 (secure 0 0 0) 0x7ffa2401a430 con 0x7ffa28107080 2024-01-24T14:19:48.703 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:45.705+0000 7ffa2fabc700 1 -- 172.21.15.19:0/1919501821 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa2810a700 con 0x7ffa28107080 2024-01-24T14:19:48.703 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:45.705+0000 7ffa1affd700 1 --2- 172.21.15.19:0/1919501821 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7ffa140383c0 0x7ffa1403a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:48.703 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:45.706+0000 7ffa1affd700 1 -- 172.21.15.19:0/1919501821 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ffa2404c4e0 con 0x7ffa28107080 2024-01-24T14:19:48.703 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:45.706+0000 7ffa2d057700 1 --2- 172.21.15.19:0/1919501821 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7ffa140383c0 0x7ffa1403a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:48.703 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:45.706+0000 7ffa2d057700 1 --2- 172.21.15.19:0/1919501821 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7ffa140383c0 0x7ffa1403a880 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c006fd0 tx=0x7ffa1c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:48.703 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:45.711+0000 7ffa1affd700 1 -- 172.21.15.19:0/1919501821 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7ffa2402d9a0 con 0x7ffa28107080 2024-01-24T14:19:48.703 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:45.920+0000 7ffa2fabc700 1 -- 172.21.15.19:0/1919501821 --> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi019", "addr": "172.21.15.19", "target": ["mon-mgr", ""]}) v1 -- 0x7ffa28101ec0 con 0x7ffa140383c0 2024-01-24T14:19:48.703 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:48.385+0000 7ffa1affd700 1 -- 172.21.15.19:0/1919501821 <== mgr.14120 v2:172.21.15.19:6800/3758061162 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7ffa28101ec0 con 0x7ffa140383c0 2024-01-24T14:19:48.703 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:48.387+0000 7ffa2fabc700 1 -- 172.21.15.19:0/1919501821 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7ffa140383c0 msgr2=0x7ffa1403a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:48.704 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:48.387+0000 7ffa2fabc700 1 --2- 172.21.15.19:0/1919501821 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7ffa140383c0 0x7ffa1403a880 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c006fd0 tx=0x7ffa1c006e40 comp rx=0 tx=0).stop 2024-01-24T14:19:48.704 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:48.387+0000 7ffa2fabc700 1 -- 172.21.15.19:0/1919501821 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffa28107080 msgr2=0x7ffa281069e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:48.704 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:48.387+0000 7ffa2fabc700 1 --2- 172.21.15.19:0/1919501821 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffa28107080 0x7ffa281069e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ffa24009130 tx=0x7ffa24011ad0 comp rx=0 tx=0).stop 2024-01-24T14:19:48.704 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:48.388+0000 7ffa2fabc700 1 -- 172.21.15.19:0/1919501821 shutdown_connections 2024-01-24T14:19:48.704 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:48.388+0000 7ffa2fabc700 1 --2- 172.21.15.19:0/1919501821 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7ffa140383c0 0x7ffa1403a880 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:48.704 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:48.388+0000 7ffa2fabc700 1 --2- 172.21.15.19:0/1919501821 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ffa28107080 0x7ffa281069e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:48.704 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:48.388+0000 7ffa2fabc700 1 -- 172.21.15.19:0/1919501821 >> 172.21.15.19:0/1919501821 conn(0x7ffa28100a70 msgr2=0x7ffa281017b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:19:48.704 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:48.388+0000 7ffa2fabc700 1 -- 172.21.15.19:0/1919501821 shutdown_connections 2024-01-24T14:19:48.704 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:48.388+0000 7ffa2fabc700 1 -- 172.21.15.19:0/1919501821 wait complete. 2024-01-24T14:19:48.704 INFO:teuthology.orchestra.run.smithi019.stdout:Deploying unmanaged mon service... 2024-01-24T14:19:49.522 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:49 smithi019 ceph-mon[40663]: from='mgr.14120 172.21.15.19:0/1799968416' entity='mgr.x' 2024-01-24T14:19:49.522 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:49 smithi019 ceph-mon[40663]: Added host smithi019 2024-01-24T14:19:49.522 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:49 smithi019 ceph-mon[40663]: from='mgr.14120 172.21.15.19:0/1799968416' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:19:49.523 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:49 smithi019 ceph-mon[40663]: from='mgr.14120 172.21.15.19:0/1799968416' entity='mgr.x' 2024-01-24T14:19:49.824 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-01-24T14:19:49.824 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.069+0000 7fc029932700 1 Processor -- start 2024-01-24T14:19:49.824 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.069+0000 7fc029932700 1 -- start start 2024-01-24T14:19:49.824 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.069+0000 7fc029932700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc024106a00 0x7fc024106e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:49.824 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.069+0000 7fc029932700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0241073f0 con 0x7fc024106a00 2024-01-24T14:19:49.824 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.069+0000 7fc022ffd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc024106a00 0x7fc024106e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:49.825 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.069+0000 7fc022ffd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc024106a00 0x7fc024106e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:55680/0 (socket says 172.21.15.19:55680) 2024-01-24T14:19:49.825 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.069+0000 7fc022ffd700 1 -- 172.21.15.19:0/4157586363 learned_addr learned my addr 172.21.15.19:0/4157586363 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:19:49.825 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.070+0000 7fc022ffd700 1 -- 172.21.15.19:0/4157586363 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc024107530 con 0x7fc024106a00 2024-01-24T14:19:49.825 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.070+0000 7fc022ffd700 1 --2- 172.21.15.19:0/4157586363 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc024106a00 0x7fc024106e20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc0140098a0 tx=0x7fc014009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37c078421b2d4590 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:49.825 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.070+0000 7fc021ffb700 1 -- 172.21.15.19:0/4157586363 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc014004160 con 0x7fc024106a00 2024-01-24T14:19:49.825 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.070+0000 7fc021ffb700 1 -- 172.21.15.19:0/4157586363 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc0140042c0 con 0x7fc024106a00 2024-01-24T14:19:49.825 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.070+0000 7fc021ffb700 1 -- 172.21.15.19:0/4157586363 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc014011a00 con 0x7fc024106a00 2024-01-24T14:19:49.825 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.070+0000 7fc029932700 1 -- 172.21.15.19:0/4157586363 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc024106a00 msgr2=0x7fc024106e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:49.825 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.070+0000 7fc029932700 1 --2- 172.21.15.19:0/4157586363 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc024106a00 0x7fc024106e20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc0140098a0 tx=0x7fc014009bb0 comp rx=0 tx=0).stop 2024-01-24T14:19:49.825 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.071+0000 7fc029932700 1 -- 172.21.15.19:0/4157586363 shutdown_connections 2024-01-24T14:19:49.825 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.071+0000 7fc029932700 1 --2- 172.21.15.19:0/4157586363 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc024106a00 0x7fc024106e20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:49.825 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.071+0000 7fc029932700 1 -- 172.21.15.19:0/4157586363 >> 172.21.15.19:0/4157586363 conn(0x7fc024100190 msgr2=0x7fc0241025d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:19:49.825 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.071+0000 7fc029932700 1 -- 172.21.15.19:0/4157586363 shutdown_connections 2024-01-24T14:19:49.825 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.071+0000 7fc029932700 1 -- 172.21.15.19:0/4157586363 wait complete. 2024-01-24T14:19:49.826 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.071+0000 7fc029932700 1 Processor -- start 2024-01-24T14:19:49.826 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.072+0000 7fc029932700 1 -- start start 2024-01-24T14:19:49.826 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.072+0000 7fc029932700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc024106a00 0x7fc02419c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:49.826 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.072+0000 7fc029932700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc02419c8f0 con 0x7fc024106a00 2024-01-24T14:19:49.826 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.072+0000 7fc022ffd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc024106a00 0x7fc02419c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:49.826 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.072+0000 7fc022ffd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc024106a00 0x7fc02419c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:55684/0 (socket says 172.21.15.19:55684) 2024-01-24T14:19:49.826 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.072+0000 7fc022ffd700 1 -- 172.21.15.19:0/2793935295 learned_addr learned my addr 172.21.15.19:0/2793935295 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:19:49.826 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.072+0000 7fc022ffd700 1 -- 172.21.15.19:0/2793935295 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc014009550 con 0x7fc024106a00 2024-01-24T14:19:49.826 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.072+0000 7fc022ffd700 1 --2- 172.21.15.19:0/2793935295 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc024106a00 0x7fc02419c3b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc014006af0 tx=0x7fc014011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:49.826 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.073+0000 7fc028930700 1 -- 172.21.15.19:0/2793935295 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc014003a40 con 0x7fc024106a00 2024-01-24T14:19:49.826 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.073+0000 7fc028930700 1 -- 172.21.15.19:0/2793935295 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc014003ba0 con 0x7fc024106a00 2024-01-24T14:19:49.826 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.073+0000 7fc028930700 1 -- 172.21.15.19:0/2793935295 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc01401a700 con 0x7fc024106a00 2024-01-24T14:19:49.826 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.073+0000 7fc029932700 1 -- 172.21.15.19:0/2793935295 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc02419caf0 con 0x7fc024106a00 2024-01-24T14:19:49.826 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.073+0000 7fc029932700 1 -- 172.21.15.19:0/2793935295 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc02419cf10 con 0x7fc024106a00 2024-01-24T14:19:49.827 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.073+0000 7fc028930700 1 -- 172.21.15.19:0/2793935295 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 8) v1 ==== 45268+0+0 (secure 0 0 0) 0x7fc01401a930 con 0x7fc024106a00 2024-01-24T14:19:49.827 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.074+0000 7fc029932700 1 -- 172.21.15.19:0/2793935295 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc024062370 con 0x7fc024106a00 2024-01-24T14:19:49.827 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.074+0000 7fc028930700 1 --2- 172.21.15.19:0/2793935295 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7fc00c038410 0x7fc00c03a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:49.827 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.074+0000 7fc028930700 1 -- 172.21.15.19:0/2793935295 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fc01404c270 con 0x7fc024106a00 2024-01-24T14:19:49.827 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.074+0000 7fc0227fc700 1 --2- 172.21.15.19:0/2793935295 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7fc00c038410 0x7fc00c03a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:49.827 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.074+0000 7fc0227fc700 1 --2- 172.21.15.19:0/2793935295 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7fc00c038410 0x7fc00c03a8d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc018006fd0 tx=0x7fc018006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:49.827 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.079+0000 7fc028930700 1 -- 172.21.15.19:0/2793935295 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fc014018db0 con 0x7fc024106a00 2024-01-24T14:19:49.827 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.310+0000 7fc029932700 1 -- 172.21.15.19:0/2793935295 --> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fc0241022b0 con 0x7fc00c038410 2024-01-24T14:19:49.827 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.315+0000 7fc028930700 1 -- 172.21.15.19:0/2793935295 <== mgr.14120 v2:172.21.15.19:6800/3758061162 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fc0241022b0 con 0x7fc00c038410 2024-01-24T14:19:49.827 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.318+0000 7fc029932700 1 -- 172.21.15.19:0/2793935295 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7fc00c038410 msgr2=0x7fc00c03a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:49.827 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.318+0000 7fc029932700 1 --2- 172.21.15.19:0/2793935295 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7fc00c038410 0x7fc00c03a8d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc018006fd0 tx=0x7fc018006e40 comp rx=0 tx=0).stop 2024-01-24T14:19:49.827 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.318+0000 7fc029932700 1 -- 172.21.15.19:0/2793935295 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc024106a00 msgr2=0x7fc02419c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:49.827 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.318+0000 7fc029932700 1 --2- 172.21.15.19:0/2793935295 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc024106a00 0x7fc02419c3b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc014006af0 tx=0x7fc014011ad0 comp rx=0 tx=0).stop 2024-01-24T14:19:49.828 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.318+0000 7fc029932700 1 -- 172.21.15.19:0/2793935295 shutdown_connections 2024-01-24T14:19:49.828 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.318+0000 7fc029932700 1 --2- 172.21.15.19:0/2793935295 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7fc00c038410 0x7fc00c03a8d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:49.828 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.318+0000 7fc029932700 1 --2- 172.21.15.19:0/2793935295 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc024106a00 0x7fc02419c3b0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:49.828 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.318+0000 7fc029932700 1 -- 172.21.15.19:0/2793935295 >> 172.21.15.19:0/2793935295 conn(0x7fc024100190 msgr2=0x7fc024104f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:19:49.828 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.318+0000 7fc029932700 1 -- 172.21.15.19:0/2793935295 shutdown_connections 2024-01-24T14:19:49.828 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:49.318+0000 7fc029932700 1 -- 172.21.15.19:0/2793935295 wait complete. 2024-01-24T14:19:49.828 INFO:teuthology.orchestra.run.smithi019.stdout:Deploying unmanaged mgr service... 2024-01-24T14:19:50.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:50 smithi019 ceph-mon[40663]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:19:50.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:50 smithi019 ceph-mon[40663]: Saving service mon spec with placement count:5 2024-01-24T14:19:50.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:50 smithi019 ceph-mon[40663]: from='mgr.14120 172.21.15.19:0/1799968416' entity='mgr.x' 2024-01-24T14:19:50.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:50 smithi019 ceph-mon[40663]: from='mgr.14120 172.21.15.19:0/1799968416' entity='mgr.x' 2024-01-24T14:19:50.974 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-01-24T14:19:50.974 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.188+0000 7fd8c8a6f700 1 Processor -- start 2024-01-24T14:19:50.974 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.188+0000 7fd8c8a6f700 1 -- start start 2024-01-24T14:19:50.974 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.189+0000 7fd8c8a6f700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd8c4104e70 0x7fd8c4105290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:50.975 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.189+0000 7fd8c8a6f700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c4079e30 con 0x7fd8c4104e70 2024-01-24T14:19:50.975 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.189+0000 7fd8c259c700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd8c4104e70 0x7fd8c4105290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:50.975 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.189+0000 7fd8c259c700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd8c4104e70 0x7fd8c4105290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:55688/0 (socket says 172.21.15.19:55688) 2024-01-24T14:19:50.975 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.189+0000 7fd8c259c700 1 -- 172.21.15.19:0/2184052411 learned_addr learned my addr 172.21.15.19:0/2184052411 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:19:50.975 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.190+0000 7fd8c259c700 1 -- 172.21.15.19:0/2184052411 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8c41057d0 con 0x7fd8c4104e70 2024-01-24T14:19:50.975 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.190+0000 7fd8c259c700 1 --2- 172.21.15.19:0/2184052411 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd8c4104e70 0x7fd8c4105290 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd8b40098a0 tx=0x7fd8b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e21ea8610d0fef1 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:50.975 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.190+0000 7fd8c159a700 1 -- 172.21.15.19:0/2184052411 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd8b4004160 con 0x7fd8c4104e70 2024-01-24T14:19:50.975 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.190+0000 7fd8c159a700 1 -- 172.21.15.19:0/2184052411 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd8b40042c0 con 0x7fd8c4104e70 2024-01-24T14:19:50.975 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.190+0000 7fd8c159a700 1 -- 172.21.15.19:0/2184052411 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd8b40119b0 con 0x7fd8c4104e70 2024-01-24T14:19:50.975 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.191+0000 7fd8c8a6f700 1 -- 172.21.15.19:0/2184052411 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd8c4104e70 msgr2=0x7fd8c4105290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:50.975 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.191+0000 7fd8c8a6f700 1 --2- 172.21.15.19:0/2184052411 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd8c4104e70 0x7fd8c4105290 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd8b40098a0 tx=0x7fd8b4009bb0 comp rx=0 tx=0).stop 2024-01-24T14:19:50.975 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.191+0000 7fd8c8a6f700 1 -- 172.21.15.19:0/2184052411 shutdown_connections 2024-01-24T14:19:50.975 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.191+0000 7fd8c8a6f700 1 --2- 172.21.15.19:0/2184052411 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd8c4104e70 0x7fd8c4105290 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:50.975 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.191+0000 7fd8c8a6f700 1 -- 172.21.15.19:0/2184052411 >> 172.21.15.19:0/2184052411 conn(0x7fd8c4100a70 msgr2=0x7fd8c4102e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:19:50.976 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.191+0000 7fd8c8a6f700 1 -- 172.21.15.19:0/2184052411 shutdown_connections 2024-01-24T14:19:50.976 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.191+0000 7fd8c8a6f700 1 -- 172.21.15.19:0/2184052411 wait complete. 2024-01-24T14:19:50.976 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.192+0000 7fd8c8a6f700 1 Processor -- start 2024-01-24T14:19:50.976 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.192+0000 7fd8c8a6f700 1 -- start start 2024-01-24T14:19:50.976 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.192+0000 7fd8c8a6f700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd8c4104e70 0x7fd8c419c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:50.976 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.192+0000 7fd8c8a6f700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c419c920 con 0x7fd8c4104e70 2024-01-24T14:19:50.976 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.192+0000 7fd8c259c700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd8c4104e70 0x7fd8c419c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:50.976 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.192+0000 7fd8c259c700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd8c4104e70 0x7fd8c419c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:55704/0 (socket says 172.21.15.19:55704) 2024-01-24T14:19:50.976 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.192+0000 7fd8c259c700 1 -- 172.21.15.19:0/270625078 learned_addr learned my addr 172.21.15.19:0/270625078 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:19:50.976 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.192+0000 7fd8c259c700 1 -- 172.21.15.19:0/270625078 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b4009550 con 0x7fd8c4104e70 2024-01-24T14:19:50.976 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.193+0000 7fd8c259c700 1 --2- 172.21.15.19:0/270625078 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd8c4104e70 0x7fd8c419c3e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd8b4006af0 tx=0x7fd8b4011a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:50.976 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.193+0000 7fd8b37fe700 1 -- 172.21.15.19:0/270625078 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd8b4003a40 con 0x7fd8c4104e70 2024-01-24T14:19:50.976 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.193+0000 7fd8c8a6f700 1 -- 172.21.15.19:0/270625078 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8c419cb20 con 0x7fd8c4104e70 2024-01-24T14:19:50.977 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.193+0000 7fd8b37fe700 1 -- 172.21.15.19:0/270625078 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd8b4003ba0 con 0x7fd8c4104e70 2024-01-24T14:19:50.977 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.193+0000 7fd8b37fe700 1 -- 172.21.15.19:0/270625078 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd8b40196c0 con 0x7fd8c4104e70 2024-01-24T14:19:50.977 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.193+0000 7fd8c8a6f700 1 -- 172.21.15.19:0/270625078 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8c419cf40 con 0x7fd8c4104e70 2024-01-24T14:19:50.977 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.194+0000 7fd8c8a6f700 1 -- 172.21.15.19:0/270625078 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8c4195c90 con 0x7fd8c4104e70 2024-01-24T14:19:50.977 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.194+0000 7fd8b37fe700 1 -- 172.21.15.19:0/270625078 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 8) v1 ==== 45268+0+0 (secure 0 0 0) 0x7fd8b401a430 con 0x7fd8c4104e70 2024-01-24T14:19:50.977 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.194+0000 7fd8b37fe700 1 --2- 172.21.15.19:0/270625078 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7fd8ac038370 0x7fd8ac03a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:50.977 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.194+0000 7fd8b37fe700 1 -- 172.21.15.19:0/270625078 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fd8b404c390 con 0x7fd8c4104e70 2024-01-24T14:19:50.977 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.194+0000 7fd8c1d9b700 1 --2- 172.21.15.19:0/270625078 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7fd8ac038370 0x7fd8ac03a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:50.977 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.195+0000 7fd8c1d9b700 1 --2- 172.21.15.19:0/270625078 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7fd8ac038370 0x7fd8ac03a830 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8006fd0 tx=0x7fd8b8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:50.977 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.199+0000 7fd8b37fe700 1 -- 172.21.15.19:0/270625078 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fd8b4018350 con 0x7fd8c4104e70 2024-01-24T14:19:50.977 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.451+0000 7fd8c8a6f700 1 -- 172.21.15.19:0/270625078 --> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fd8c4061180 con 0x7fd8ac038370 2024-01-24T14:19:50.977 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.457+0000 7fd8b37fe700 1 -- 172.21.15.19:0/270625078 <== mgr.14120 v2:172.21.15.19:6800/3758061162 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fd8c4061180 con 0x7fd8ac038370 2024-01-24T14:19:50.977 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.459+0000 7fd8c8a6f700 1 -- 172.21.15.19:0/270625078 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7fd8ac038370 msgr2=0x7fd8ac03a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:50.977 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.459+0000 7fd8c8a6f700 1 --2- 172.21.15.19:0/270625078 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7fd8ac038370 0x7fd8ac03a830 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8006fd0 tx=0x7fd8b8006e40 comp rx=0 tx=0).stop 2024-01-24T14:19:50.978 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.459+0000 7fd8c8a6f700 1 -- 172.21.15.19:0/270625078 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd8c4104e70 msgr2=0x7fd8c419c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:50.978 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.459+0000 7fd8c8a6f700 1 --2- 172.21.15.19:0/270625078 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd8c4104e70 0x7fd8c419c3e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd8b4006af0 tx=0x7fd8b4011a10 comp rx=0 tx=0).stop 2024-01-24T14:19:50.978 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.459+0000 7fd8c8a6f700 1 -- 172.21.15.19:0/270625078 shutdown_connections 2024-01-24T14:19:50.978 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.459+0000 7fd8c8a6f700 1 --2- 172.21.15.19:0/270625078 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7fd8ac038370 0x7fd8ac03a830 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:50.978 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.459+0000 7fd8c8a6f700 1 --2- 172.21.15.19:0/270625078 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd8c4104e70 0x7fd8c419c3e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:50.978 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.459+0000 7fd8c8a6f700 1 -- 172.21.15.19:0/270625078 >> 172.21.15.19:0/270625078 conn(0x7fd8c4100a70 msgr2=0x7fd8c4190a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:19:50.978 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.459+0000 7fd8c8a6f700 1 -- 172.21.15.19:0/270625078 shutdown_connections 2024-01-24T14:19:50.978 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:50.459+0000 7fd8c8a6f700 1 -- 172.21.15.19:0/270625078 wait complete. 2024-01-24T14:19:51.734 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:51 smithi019 ceph-mon[40663]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:19:51.734 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:51 smithi019 ceph-mon[40663]: Saving service mgr spec with placement count:2 2024-01-24T14:19:51.734 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:51 smithi019 ceph-mon[40663]: from='mgr.14120 172.21.15.19:0/1799968416' entity='mgr.x' 2024-01-24T14:19:51.734 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:51 smithi019 ceph-mon[40663]: from='mgr.14120 172.21.15.19:0/1799968416' entity='mgr.x' 2024-01-24T14:19:51.889 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.334+0000 7f466b22d700 1 Processor -- start 2024-01-24T14:19:51.889 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.335+0000 7f466b22d700 1 -- start start 2024-01-24T14:19:51.889 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.335+0000 7f466b22d700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4664107060 0x7f4664109450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:51.889 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.335+0000 7f466b22d700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4664079e50 con 0x7f4664107060 2024-01-24T14:19:51.889 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.335+0000 7f4668fc9700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4664107060 0x7f4664109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:51.889 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.335+0000 7f4668fc9700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4664107060 0x7f4664109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:55714/0 (socket says 172.21.15.19:55714) 2024-01-24T14:19:51.889 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.335+0000 7f4668fc9700 1 -- 172.21.15.19:0/3074424460 learned_addr learned my addr 172.21.15.19:0/3074424460 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:19:51.889 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.335+0000 7f4668fc9700 1 -- 172.21.15.19:0/3074424460 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4664109990 con 0x7f4664107060 2024-01-24T14:19:51.889 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.336+0000 7f4668fc9700 1 --2- 172.21.15.19:0/3074424460 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4664107060 0x7f4664109450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f46580098a0 tx=0x7f4658009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ccf0f9388d4c367 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:51.889 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.336+0000 7f46637fe700 1 -- 172.21.15.19:0/3074424460 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4658004160 con 0x7f4664107060 2024-01-24T14:19:51.890 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.336+0000 7f46637fe700 1 -- 172.21.15.19:0/3074424460 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f46580042c0 con 0x7f4664107060 2024-01-24T14:19:51.890 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.336+0000 7f46637fe700 1 -- 172.21.15.19:0/3074424460 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f46580119b0 con 0x7f4664107060 2024-01-24T14:19:51.890 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.336+0000 7f466b22d700 1 -- 172.21.15.19:0/3074424460 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4664107060 msgr2=0x7f4664109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:51.890 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.336+0000 7f466b22d700 1 --2- 172.21.15.19:0/3074424460 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4664107060 0x7f4664109450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f46580098a0 tx=0x7f4658009bb0 comp rx=0 tx=0).stop 2024-01-24T14:19:51.890 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.336+0000 7f466b22d700 1 -- 172.21.15.19:0/3074424460 shutdown_connections 2024-01-24T14:19:51.890 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.336+0000 7f466b22d700 1 --2- 172.21.15.19:0/3074424460 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4664107060 0x7f4664109450 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:51.890 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.336+0000 7f466b22d700 1 -- 172.21.15.19:0/3074424460 >> 172.21.15.19:0/3074424460 conn(0x7f4664100a70 msgr2=0x7f4664102e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:19:51.890 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.336+0000 7f466b22d700 1 -- 172.21.15.19:0/3074424460 shutdown_connections 2024-01-24T14:19:51.890 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.337+0000 7f466b22d700 1 -- 172.21.15.19:0/3074424460 wait complete. 2024-01-24T14:19:51.890 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.337+0000 7f466b22d700 1 Processor -- start 2024-01-24T14:19:51.890 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.337+0000 7f466b22d700 1 -- start start 2024-01-24T14:19:51.890 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.337+0000 7f466b22d700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4664107060 0x7f46641069c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:51.890 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.337+0000 7f466b22d700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4664106f00 con 0x7f4664107060 2024-01-24T14:19:51.891 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.337+0000 7f4668fc9700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4664107060 0x7f46641069c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:51.891 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.337+0000 7f4668fc9700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4664107060 0x7f46641069c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:55716/0 (socket says 172.21.15.19:55716) 2024-01-24T14:19:51.891 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.338+0000 7f4668fc9700 1 -- 172.21.15.19:0/2997404156 learned_addr learned my addr 172.21.15.19:0/2997404156 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:19:51.891 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.338+0000 7f4668fc9700 1 -- 172.21.15.19:0/2997404156 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4658009550 con 0x7f4664107060 2024-01-24T14:19:51.891 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.338+0000 7f4668fc9700 1 --2- 172.21.15.19:0/2997404156 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4664107060 0x7f46641069c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4658006af0 tx=0x7f4658011a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:51.891 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.338+0000 7f4661ffb700 1 -- 172.21.15.19:0/2997404156 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4658003a40 con 0x7f4664107060 2024-01-24T14:19:51.891 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.338+0000 7f4661ffb700 1 -- 172.21.15.19:0/2997404156 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4658003ba0 con 0x7f4664107060 2024-01-24T14:19:51.891 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.338+0000 7f466b22d700 1 -- 172.21.15.19:0/2997404156 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4664105010 con 0x7f4664107060 2024-01-24T14:19:51.891 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.338+0000 7f466b22d700 1 -- 172.21.15.19:0/2997404156 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4664105430 con 0x7f4664107060 2024-01-24T14:19:51.891 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.338+0000 7f4661ffb700 1 -- 172.21.15.19:0/2997404156 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f46580196c0 con 0x7f4664107060 2024-01-24T14:19:51.891 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.339+0000 7f466b22d700 1 -- 172.21.15.19:0/2997404156 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4648005320 con 0x7f4664107060 2024-01-24T14:19:51.891 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.340+0000 7f4661ffb700 1 -- 172.21.15.19:0/2997404156 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 8) v1 ==== 45268+0+0 (secure 0 0 0) 0x7f4658019980 con 0x7f4664107060 2024-01-24T14:19:51.891 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.340+0000 7f4661ffb700 1 --2- 172.21.15.19:0/2997404156 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7f4654038370 0x7f465403a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:51.891 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.340+0000 7f4661ffb700 1 -- 172.21.15.19:0/2997404156 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f465804c840 con 0x7f4664107060 2024-01-24T14:19:51.892 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.340+0000 7f4663fff700 1 --2- 172.21.15.19:0/2997404156 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7f4654038370 0x7f465403a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:51.892 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.341+0000 7f4663fff700 1 --2- 172.21.15.19:0/2997404156 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7f4654038370 0x7f465403a830 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4650006fd0 tx=0x7f4650006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:51.892 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.345+0000 7f4661ffb700 1 -- 172.21.15.19:0/2997404156 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f4658023390 con 0x7f4664107060 2024-01-24T14:19:51.892 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.570+0000 7f466b22d700 1 -- 172.21.15.19:0/2997404156 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f4648005190 con 0x7f4664107060 2024-01-24T14:19:51.892 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.576+0000 7f4661ffb700 1 -- 172.21.15.19:0/2997404156 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f465804b0d0 con 0x7f4664107060 2024-01-24T14:19:51.892 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.579+0000 7f466b22d700 1 -- 172.21.15.19:0/2997404156 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7f4654038370 msgr2=0x7f465403a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:51.892 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.579+0000 7f466b22d700 1 --2- 172.21.15.19:0/2997404156 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7f4654038370 0x7f465403a830 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4650006fd0 tx=0x7f4650006e40 comp rx=0 tx=0).stop 2024-01-24T14:19:51.892 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.579+0000 7f466b22d700 1 -- 172.21.15.19:0/2997404156 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4664107060 msgr2=0x7f46641069c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:51.892 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.579+0000 7f466b22d700 1 --2- 172.21.15.19:0/2997404156 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4664107060 0x7f46641069c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4658006af0 tx=0x7f4658011a10 comp rx=0 tx=0).stop 2024-01-24T14:19:51.892 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.579+0000 7f466b22d700 1 -- 172.21.15.19:0/2997404156 shutdown_connections 2024-01-24T14:19:51.892 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.579+0000 7f466b22d700 1 --2- 172.21.15.19:0/2997404156 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7f4654038370 0x7f465403a830 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:51.892 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.579+0000 7f466b22d700 1 --2- 172.21.15.19:0/2997404156 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4664107060 0x7f46641069c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:51.892 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.579+0000 7f466b22d700 1 -- 172.21.15.19:0/2997404156 >> 172.21.15.19:0/2997404156 conn(0x7f4664100a70 msgr2=0x7f46641017e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:19:51.893 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.579+0000 7f466b22d700 1 -- 172.21.15.19:0/2997404156 shutdown_connections 2024-01-24T14:19:51.893 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:51.579+0000 7f466b22d700 1 -- 172.21.15.19:0/2997404156 wait complete. 2024-01-24T14:19:52.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:52 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/2997404156' entity='client.admin' 2024-01-24T14:19:52.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:52 smithi019 ceph-mon[40663]: from='mgr.14120 172.21.15.19:0/1799968416' entity='mgr.x' 2024-01-24T14:19:52.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:52 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/673212390' entity='client.admin' 2024-01-24T14:19:52.983 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.253+0000 7facf897e700 1 Processor -- start 2024-01-24T14:19:52.984 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.253+0000 7facf897e700 1 -- start start 2024-01-24T14:19:52.984 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.253+0000 7facf897e700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7facf4107000 0x7facf41093f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:52.984 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.253+0000 7facf897e700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf4079eb0 con 0x7facf4107000 2024-01-24T14:19:52.984 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.254+0000 7facf259c700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7facf4107000 0x7facf41093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:52.984 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.254+0000 7facf259c700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7facf4107000 0x7facf41093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:43624/0 (socket says 172.21.15.19:43624) 2024-01-24T14:19:52.984 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.254+0000 7facf259c700 1 -- 172.21.15.19:0/3668094187 learned_addr learned my addr 172.21.15.19:0/3668094187 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:19:52.984 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.254+0000 7facf259c700 1 -- 172.21.15.19:0/3668094187 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf4109930 con 0x7facf4107000 2024-01-24T14:19:52.984 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.255+0000 7facf259c700 1 --2- 172.21.15.19:0/3668094187 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7facf4107000 0x7facf41093f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7facdc0098a0 tx=0x7facdc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30fda0f0f2ac8e37 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:52.984 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.255+0000 7facf159a700 1 -- 172.21.15.19:0/3668094187 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7facdc004160 con 0x7facf4107000 2024-01-24T14:19:52.984 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.255+0000 7facf159a700 1 -- 172.21.15.19:0/3668094187 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7facdc0042c0 con 0x7facf4107000 2024-01-24T14:19:52.984 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.255+0000 7facf159a700 1 -- 172.21.15.19:0/3668094187 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7facdc0119b0 con 0x7facf4107000 2024-01-24T14:19:52.984 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.256+0000 7facf897e700 1 -- 172.21.15.19:0/3668094187 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7facf4107000 msgr2=0x7facf41093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:52.985 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.256+0000 7facf897e700 1 --2- 172.21.15.19:0/3668094187 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7facf4107000 0x7facf41093f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7facdc0098a0 tx=0x7facdc009bb0 comp rx=0 tx=0).stop 2024-01-24T14:19:52.985 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.256+0000 7facf897e700 1 -- 172.21.15.19:0/3668094187 shutdown_connections 2024-01-24T14:19:52.985 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.256+0000 7facf897e700 1 --2- 172.21.15.19:0/3668094187 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7facf4107000 0x7facf41093f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:52.985 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.256+0000 7facf897e700 1 -- 172.21.15.19:0/3668094187 >> 172.21.15.19:0/3668094187 conn(0x7facf4100a70 msgr2=0x7facf4102e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:19:52.985 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.256+0000 7facf897e700 1 -- 172.21.15.19:0/3668094187 shutdown_connections 2024-01-24T14:19:52.985 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.256+0000 7facf897e700 1 -- 172.21.15.19:0/3668094187 wait complete. 2024-01-24T14:19:52.985 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.257+0000 7facf897e700 1 Processor -- start 2024-01-24T14:19:52.985 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.257+0000 7facf897e700 1 -- start start 2024-01-24T14:19:52.985 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.257+0000 7facf897e700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7facf4107000 0x7facf419a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:52.985 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.257+0000 7facf897e700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf419a700 con 0x7facf4107000 2024-01-24T14:19:52.985 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.257+0000 7facf259c700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7facf4107000 0x7facf419a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:52.985 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.257+0000 7facf259c700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7facf4107000 0x7facf419a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:43630/0 (socket says 172.21.15.19:43630) 2024-01-24T14:19:52.985 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.257+0000 7facf259c700 1 -- 172.21.15.19:0/673212390 learned_addr learned my addr 172.21.15.19:0/673212390 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:19:52.985 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.257+0000 7facf259c700 1 -- 172.21.15.19:0/673212390 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facdc009550 con 0x7facf4107000 2024-01-24T14:19:52.985 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.258+0000 7facf259c700 1 --2- 172.21.15.19:0/673212390 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7facf4107000 0x7facf419a1c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7facdc003f00 tx=0x7facdc011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:52.986 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.258+0000 7faceb7fe700 1 -- 172.21.15.19:0/673212390 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7facdc003b20 con 0x7facf4107000 2024-01-24T14:19:52.986 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.258+0000 7faceb7fe700 1 -- 172.21.15.19:0/673212390 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7facdc003c80 con 0x7facf4107000 2024-01-24T14:19:52.986 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.258+0000 7facf897e700 1 -- 172.21.15.19:0/673212390 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facf419a900 con 0x7facf4107000 2024-01-24T14:19:52.986 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.258+0000 7faceb7fe700 1 -- 172.21.15.19:0/673212390 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7facdc019770 con 0x7facf4107000 2024-01-24T14:19:52.986 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.258+0000 7facf897e700 1 -- 172.21.15.19:0/673212390 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facf419ad20 con 0x7facf4107000 2024-01-24T14:19:52.986 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.259+0000 7faceb7fe700 1 -- 172.21.15.19:0/673212390 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 8) v1 ==== 45268+0+0 (secure 0 0 0) 0x7facdc0198f0 con 0x7facf4107000 2024-01-24T14:19:52.986 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.259+0000 7faceb7fe700 1 --2- 172.21.15.19:0/673212390 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7face0038410 0x7face003a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:52.986 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.259+0000 7faceb7fe700 1 -- 172.21.15.19:0/673212390 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7facdc04c4b0 con 0x7facf4107000 2024-01-24T14:19:52.986 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.259+0000 7facf897e700 1 -- 172.21.15.19:0/673212390 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facf410a770 con 0x7facf4107000 2024-01-24T14:19:52.986 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.259+0000 7facf1d9b700 1 --2- 172.21.15.19:0/673212390 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7face0038410 0x7face003a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:52.986 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.260+0000 7facf1d9b700 1 --2- 172.21.15.19:0/673212390 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7face0038410 0x7face003a8d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7face4006fd0 tx=0x7face4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:52.986 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.266+0000 7faceb7fe700 1 -- 172.21.15.19:0/673212390 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7facdc018db0 con 0x7facf4107000 2024-01-24T14:19:52.986 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.483+0000 7facf897e700 1 -- 172.21.15.19:0/673212390 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7facf419dd70 con 0x7facf4107000 2024-01-24T14:19:52.987 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.486+0000 7faceb7fe700 1 -- 172.21.15.19:0/673212390 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7facdc018350 con 0x7facf4107000 2024-01-24T14:19:52.987 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.489+0000 7facf897e700 1 -- 172.21.15.19:0/673212390 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7face0038410 msgr2=0x7face003a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:52.987 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.489+0000 7facf897e700 1 --2- 172.21.15.19:0/673212390 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7face0038410 0x7face003a8d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7face4006fd0 tx=0x7face4006e40 comp rx=0 tx=0).stop 2024-01-24T14:19:52.987 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.489+0000 7facf897e700 1 -- 172.21.15.19:0/673212390 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7facf4107000 msgr2=0x7facf419a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:52.987 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.489+0000 7facf897e700 1 --2- 172.21.15.19:0/673212390 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7facf4107000 0x7facf419a1c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7facdc003f00 tx=0x7facdc011a30 comp rx=0 tx=0).stop 2024-01-24T14:19:52.987 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.489+0000 7facf897e700 1 -- 172.21.15.19:0/673212390 shutdown_connections 2024-01-24T14:19:52.987 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.489+0000 7facf897e700 1 --2- 172.21.15.19:0/673212390 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7face0038410 0x7face003a8d0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:52.987 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.489+0000 7facf897e700 1 --2- 172.21.15.19:0/673212390 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7facf4107000 0x7facf419a1c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:52.987 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.489+0000 7facf897e700 1 -- 172.21.15.19:0/673212390 >> 172.21.15.19:0/673212390 conn(0x7facf4100a70 msgr2=0x7facf41017f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:19:52.987 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.489+0000 7facf897e700 1 -- 172.21.15.19:0/673212390 shutdown_connections 2024-01-24T14:19:52.987 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:52.489+0000 7facf897e700 1 -- 172.21.15.19:0/673212390 wait complete. 2024-01-24T14:19:52.987 INFO:teuthology.orchestra.run.smithi019.stdout:Enabling the dashboard module... 2024-01-24T14:19:55.670 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:55 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/1691839979' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-01-24T14:19:55.671 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:55 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: ignoring --setuser ceph since I am not root 2024-01-24T14:19:55.671 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:55 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: ignoring --setgroup ceph since I am not root 2024-01-24T14:19:55.732 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:54.153+0000 7f4145401700 1 Processor -- start 2024-01-24T14:19:55.732 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:54.153+0000 7f4145401700 1 -- start start 2024-01-24T14:19:55.732 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:54.153+0000 7f4145401700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4140107070 0x7f4140109460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:55.732 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:54.153+0000 7f4145401700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4140079e30 con 0x7f4140107070 2024-01-24T14:19:55.732 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:54.153+0000 7f413effd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4140107070 0x7f4140109460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:55.732 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:54.153+0000 7f413effd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4140107070 0x7f4140109460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:43638/0 (socket says 172.21.15.19:43638) 2024-01-24T14:19:55.732 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:54.153+0000 7f413effd700 1 -- 172.21.15.19:0/1951425621 learned_addr learned my addr 172.21.15.19:0/1951425621 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:19:55.732 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:54.154+0000 7f413effd700 1 -- 172.21.15.19:0/1951425621 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41401099a0 con 0x7f4140107070 2024-01-24T14:19:55.733 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:54.154+0000 7f413effd700 1 --2- 172.21.15.19:0/1951425621 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4140107070 0x7f4140109460 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f41300098a0 tx=0x7f4130009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7112fe8d67f2f63 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:55.733 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:54.154+0000 7f413dffb700 1 -- 172.21.15.19:0/1951425621 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4130004160 con 0x7f4140107070 2024-01-24T14:19:55.733 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:54.154+0000 7f413dffb700 1 -- 172.21.15.19:0/1951425621 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f41300042c0 con 0x7f4140107070 2024-01-24T14:19:55.733 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:54.154+0000 7f413dffb700 1 -- 172.21.15.19:0/1951425621 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f41300119b0 con 0x7f4140107070 2024-01-24T14:19:55.733 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:54.154+0000 7f4145401700 1 -- 172.21.15.19:0/1951425621 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4140107070 msgr2=0x7f4140109460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:55.733 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:54.154+0000 7f4145401700 1 --2- 172.21.15.19:0/1951425621 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4140107070 0x7f4140109460 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f41300098a0 tx=0x7f4130009bb0 comp rx=0 tx=0).stop 2024-01-24T14:19:55.733 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:54.155+0000 7f4145401700 1 -- 172.21.15.19:0/1951425621 shutdown_connections 2024-01-24T14:19:55.733 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:54.155+0000 7f4145401700 1 --2- 172.21.15.19:0/1951425621 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4140107070 0x7f4140109460 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:55.733 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:54.155+0000 7f4145401700 1 -- 172.21.15.19:0/1951425621 >> 172.21.15.19:0/1951425621 conn(0x7f4140100a40 msgr2=0x7f4140102e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:19:55.733 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:54.155+0000 7f4145401700 1 -- 172.21.15.19:0/1951425621 shutdown_connections 2024-01-24T14:19:55.733 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:54.155+0000 7f4145401700 1 -- 172.21.15.19:0/1951425621 wait complete. 2024-01-24T14:19:55.733 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:54.155+0000 7f4145401700 1 Processor -- start 2024-01-24T14:19:55.733 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:54.156+0000 7f4145401700 1 -- start start 2024-01-24T14:19:55.734 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:54.156+0000 7f4145401700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4140107070 0x7f414019c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:55.734 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:54.156+0000 7f4145401700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f414019c8e0 con 0x7f4140107070 2024-01-24T14:19:55.734 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:54.156+0000 7f413effd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4140107070 0x7f414019c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:55.734 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:54.156+0000 7f413effd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4140107070 0x7f414019c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:43654/0 (socket says 172.21.15.19:43654) 2024-01-24T14:19:55.734 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:54.156+0000 7f413effd700 1 -- 172.21.15.19:0/1691839979 learned_addr learned my addr 172.21.15.19:0/1691839979 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:19:55.734 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:54.156+0000 7f413effd700 1 -- 172.21.15.19:0/1691839979 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4130009550 con 0x7f4140107070 2024-01-24T14:19:55.734 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:54.156+0000 7f413effd700 1 --2- 172.21.15.19:0/1691839979 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4140107070 0x7f414019c3a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4130006af0 tx=0x7f4130011a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:55.734 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:54.157+0000 7f4127fff700 1 -- 172.21.15.19:0/1691839979 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4130003a40 con 0x7f4140107070 2024-01-24T14:19:55.734 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:54.157+0000 7f4127fff700 1 -- 172.21.15.19:0/1691839979 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4130003ba0 con 0x7f4140107070 2024-01-24T14:19:55.734 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:54.157+0000 7f4145401700 1 -- 172.21.15.19:0/1691839979 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f414019cae0 con 0x7f4140107070 2024-01-24T14:19:55.734 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:54.157+0000 7f4127fff700 1 -- 172.21.15.19:0/1691839979 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f41300196c0 con 0x7f4140107070 2024-01-24T14:19:55.734 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:54.157+0000 7f4145401700 1 -- 172.21.15.19:0/1691839979 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f414019cf00 con 0x7f4140107070 2024-01-24T14:19:55.734 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:54.158+0000 7f4127fff700 1 -- 172.21.15.19:0/1691839979 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 8) v1 ==== 45268+0+0 (secure 0 0 0) 0x7f4130019980 con 0x7f4140107070 2024-01-24T14:19:55.734 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:54.158+0000 7f4145401700 1 -- 172.21.15.19:0/1691839979 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4140062370 con 0x7f4140107070 2024-01-24T14:19:55.734 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:54.158+0000 7f4127fff700 1 --2- 172.21.15.19:0/1691839979 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7f4128038370 0x7f412803a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:55.735 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:54.158+0000 7f4127fff700 1 -- 172.21.15.19:0/1691839979 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f413004c1b0 con 0x7f4140107070 2024-01-24T14:19:55.735 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:54.158+0000 7f413e7fc700 1 --2- 172.21.15.19:0/1691839979 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7f4128038370 0x7f412803a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:55.735 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:54.158+0000 7f413e7fc700 1 --2- 172.21.15.19:0/1691839979 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7f4128038370 0x7f412803a830 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4134006fd0 tx=0x7f4134006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:55.735 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:54.163+0000 7f4127fff700 1 -- 172.21.15.19:0/1691839979 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f413002d430 con 0x7f4140107070 2024-01-24T14:19:55.735 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:54.433+0000 7f4145401700 1 -- 172.21.15.19:0/1691839979 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f414019fff0 con 0x7f4140107070 2024-01-24T14:19:55.735 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:55.362+0000 7f4127fff700 1 -- 172.21.15.19:0/1691839979 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mgrmap(e 9) v1 ==== 45281+0+0 (secure 0 0 0) 0x7f413000ed40 con 0x7f4140107070 2024-01-24T14:19:55.735 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:55.364+0000 7f4127fff700 1 -- 172.21.15.19:0/1691839979 <== mon.0 v2:172.21.15.19:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f413004bf10 con 0x7f4140107070 2024-01-24T14:19:55.735 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:55.367+0000 7f4145401700 1 -- 172.21.15.19:0/1691839979 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7f4128038370 msgr2=0x7f412803a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:55.735 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:55.367+0000 7f4145401700 1 --2- 172.21.15.19:0/1691839979 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7f4128038370 0x7f412803a830 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4134006fd0 tx=0x7f4134006e40 comp rx=0 tx=0).stop 2024-01-24T14:19:55.735 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:55.367+0000 7f4145401700 1 -- 172.21.15.19:0/1691839979 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4140107070 msgr2=0x7f414019c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:55.735 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:55.367+0000 7f4145401700 1 --2- 172.21.15.19:0/1691839979 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4140107070 0x7f414019c3a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4130006af0 tx=0x7f4130011a10 comp rx=0 tx=0).stop 2024-01-24T14:19:55.735 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:55.367+0000 7f4145401700 1 -- 172.21.15.19:0/1691839979 shutdown_connections 2024-01-24T14:19:55.735 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:55.367+0000 7f4145401700 1 --2- 172.21.15.19:0/1691839979 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7f4128038370 0x7f412803a830 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:55.735 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:55.367+0000 7f4145401700 1 --2- 172.21.15.19:0/1691839979 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4140107070 0x7f414019c3a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:55.736 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:55.367+0000 7f4145401700 1 -- 172.21.15.19:0/1691839979 >> 172.21.15.19:0/1691839979 conn(0x7f4140100a40 msgr2=0x7f4140101770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:19:55.736 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:55.367+0000 7f4145401700 1 -- 172.21.15.19:0/1691839979 shutdown_connections 2024-01-24T14:19:55.736 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:55.367+0000 7f4145401700 1 -- 172.21.15.19:0/1691839979 wait complete. 2024-01-24T14:19:55.970 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:55 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:55.864+0000 7f99dc943200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-24T14:19:56.235 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:55 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:55.970+0000 7f99dc943200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-24T14:19:56.235 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:56 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:56.072+0000 7f99dc943200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-24T14:19:56.235 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:56 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:56.170+0000 7f99dc943200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-24T14:19:56.712 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:56 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/1691839979' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-01-24T14:19:56.712 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:56 smithi019 ceph-mon[40663]: mgrmap e9: x(active, since 15s) 2024-01-24T14:19:56.765 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout { 2024-01-24T14:19:56.765 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-01-24T14:19:56.765 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "available": true, 2024-01-24T14:19:56.765 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-01-24T14:19:56.765 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-01-24T14:19:56.766 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout } 2024-01-24T14:19:56.766 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.122+0000 7fc17254e700 1 Processor -- start 2024-01-24T14:19:56.766 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.122+0000 7fc17254e700 1 -- start start 2024-01-24T14:19:56.766 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.122+0000 7fc17254e700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc16c07bec0 0x7fc16c07a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:56.766 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.122+0000 7fc17254e700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc16c07a8f0 con 0x7fc16c07bec0 2024-01-24T14:19:56.766 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.123+0000 7fc16bfff700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc16c07bec0 0x7fc16c07a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:56.766 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.123+0000 7fc16bfff700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc16c07bec0 0x7fc16c07a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:43680/0 (socket says 172.21.15.19:43680) 2024-01-24T14:19:56.766 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.123+0000 7fc16bfff700 1 -- 172.21.15.19:0/1653482701 learned_addr learned my addr 172.21.15.19:0/1653482701 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:19:56.766 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.123+0000 7fc16bfff700 1 -- 172.21.15.19:0/1653482701 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc16c07aa30 con 0x7fc16c07bec0 2024-01-24T14:19:56.766 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.123+0000 7fc16bfff700 1 --2- 172.21.15.19:0/1653482701 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc16c07bec0 0x7fc16c07a320 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc15c0098a0 tx=0x7fc15c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3c49dce339960d8 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:56.766 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.123+0000 7fc16affd700 1 -- 172.21.15.19:0/1653482701 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc15c004160 con 0x7fc16c07bec0 2024-01-24T14:19:56.766 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.123+0000 7fc16affd700 1 -- 172.21.15.19:0/1653482701 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc15c0042c0 con 0x7fc16c07bec0 2024-01-24T14:19:56.766 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.123+0000 7fc16affd700 1 -- 172.21.15.19:0/1653482701 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc15c0119b0 con 0x7fc16c07bec0 2024-01-24T14:19:56.766 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.124+0000 7fc17254e700 1 -- 172.21.15.19:0/1653482701 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc16c07bec0 msgr2=0x7fc16c07a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:56.766 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.124+0000 7fc17254e700 1 --2- 172.21.15.19:0/1653482701 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc16c07bec0 0x7fc16c07a320 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc15c0098a0 tx=0x7fc15c009bb0 comp rx=0 tx=0).stop 2024-01-24T14:19:56.767 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.125+0000 7fc17254e700 1 -- 172.21.15.19:0/1653482701 shutdown_connections 2024-01-24T14:19:56.767 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.125+0000 7fc17254e700 1 --2- 172.21.15.19:0/1653482701 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc16c07bec0 0x7fc16c07a320 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:56.767 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.125+0000 7fc17254e700 1 -- 172.21.15.19:0/1653482701 >> 172.21.15.19:0/1653482701 conn(0x7fc16c101b60 msgr2=0x7fc16c103f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:19:56.767 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.125+0000 7fc17254e700 1 -- 172.21.15.19:0/1653482701 shutdown_connections 2024-01-24T14:19:56.767 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.125+0000 7fc17254e700 1 -- 172.21.15.19:0/1653482701 wait complete. 2024-01-24T14:19:56.767 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.125+0000 7fc17254e700 1 Processor -- start 2024-01-24T14:19:56.767 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.125+0000 7fc17254e700 1 -- start start 2024-01-24T14:19:56.767 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.126+0000 7fc17254e700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc16c07bec0 0x7fc16c19c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:56.767 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.126+0000 7fc17254e700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc16c19c940 con 0x7fc16c07bec0 2024-01-24T14:19:56.767 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.126+0000 7fc16bfff700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc16c07bec0 0x7fc16c19c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:19:56.767 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.126+0000 7fc16bfff700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc16c07bec0 0x7fc16c19c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:43690/0 (socket says 172.21.15.19:43690) 2024-01-24T14:19:56.767 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.126+0000 7fc16bfff700 1 -- 172.21.15.19:0/1072425117 learned_addr learned my addr 172.21.15.19:0/1072425117 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:19:56.767 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.126+0000 7fc16bfff700 1 -- 172.21.15.19:0/1072425117 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc15c009550 con 0x7fc16c07bec0 2024-01-24T14:19:56.767 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.126+0000 7fc16bfff700 1 --2- 172.21.15.19:0/1072425117 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc16c07bec0 0x7fc16c19c400 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc15c006af0 tx=0x7fc15c011a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:19:56.768 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.126+0000 7fc1697fa700 1 -- 172.21.15.19:0/1072425117 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc15c003a40 con 0x7fc16c07bec0 2024-01-24T14:19:56.768 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.127+0000 7fc1697fa700 1 -- 172.21.15.19:0/1072425117 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc15c003ba0 con 0x7fc16c07bec0 2024-01-24T14:19:56.768 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.127+0000 7fc17254e700 1 -- 172.21.15.19:0/1072425117 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc16c19cb40 con 0x7fc16c07bec0 2024-01-24T14:19:56.768 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.127+0000 7fc1697fa700 1 -- 172.21.15.19:0/1072425117 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc15c019690 con 0x7fc16c07bec0 2024-01-24T14:19:56.768 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.127+0000 7fc17254e700 1 -- 172.21.15.19:0/1072425117 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc16c19cf60 con 0x7fc16c07bec0 2024-01-24T14:19:56.768 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.127+0000 7fc1697fa700 1 -- 172.21.15.19:0/1072425117 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 9) v1 ==== 45281+0+0 (secure 0 0 0) 0x7fc15c019900 con 0x7fc16c07bec0 2024-01-24T14:19:56.768 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.128+0000 7fc1697fa700 1 --2- 172.21.15.19:0/1072425117 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7fc154038410 0x7fc15403a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:19:56.768 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.128+0000 7fc1697fa700 1 -- 172.21.15.19:0/1072425117 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fc15c04d0c0 con 0x7fc16c07bec0 2024-01-24T14:19:56.768 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.128+0000 7fc16b7fe700 1 -- 172.21.15.19:0/1072425117 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7fc154038410 msgr2=0x7fc15403a8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:6800/3758061162 2024-01-24T14:19:56.768 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.128+0000 7fc16b7fe700 1 --2- 172.21.15.19:0/1072425117 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7fc154038410 0x7fc15403a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-24T14:19:56.768 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.128+0000 7fc17254e700 1 -- 172.21.15.19:0/1072425117 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc16c062370 con 0x7fc16c07bec0 2024-01-24T14:19:56.768 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.133+0000 7fc1697fa700 1 -- 172.21.15.19:0/1072425117 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fc15c0243f0 con 0x7fc16c07bec0 2024-01-24T14:19:56.768 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.329+0000 7fc16b7fe700 1 -- 172.21.15.19:0/1072425117 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7fc154038410 msgr2=0x7fc15403a8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:6800/3758061162 2024-01-24T14:19:56.769 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.329+0000 7fc16b7fe700 1 --2- 172.21.15.19:0/1072425117 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7fc154038410 0x7fc15403a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-24T14:19:56.769 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.400+0000 7fc17254e700 1 -- 172.21.15.19:0/1072425117 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fc16c19fa90 con 0x7fc16c07bec0 2024-01-24T14:19:56.769 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.400+0000 7fc1697fa700 1 -- 172.21.15.19:0/1072425117 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7fc15c04a030 con 0x7fc16c07bec0 2024-01-24T14:19:56.769 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.402+0000 7fc17254e700 1 -- 172.21.15.19:0/1072425117 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7fc154038410 msgr2=0x7fc15403a8d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-24T14:19:56.769 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.402+0000 7fc17254e700 1 --2- 172.21.15.19:0/1072425117 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7fc154038410 0x7fc15403a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:56.769 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.402+0000 7fc17254e700 1 -- 172.21.15.19:0/1072425117 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc16c07bec0 msgr2=0x7fc16c19c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:19:56.769 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.402+0000 7fc17254e700 1 --2- 172.21.15.19:0/1072425117 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc16c07bec0 0x7fc16c19c400 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc15c006af0 tx=0x7fc15c011a10 comp rx=0 tx=0).stop 2024-01-24T14:19:56.769 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.403+0000 7fc17254e700 1 -- 172.21.15.19:0/1072425117 shutdown_connections 2024-01-24T14:19:56.769 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.403+0000 7fc17254e700 1 --2- 172.21.15.19:0/1072425117 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7fc154038410 0x7fc15403a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:56.769 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.403+0000 7fc17254e700 1 --2- 172.21.15.19:0/1072425117 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc16c07bec0 0x7fc16c19c400 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:19:56.769 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.403+0000 7fc17254e700 1 -- 172.21.15.19:0/1072425117 >> 172.21.15.19:0/1072425117 conn(0x7fc16c101b60 msgr2=0x7fc16c1028f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:19:56.769 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.403+0000 7fc17254e700 1 -- 172.21.15.19:0/1072425117 shutdown_connections 2024-01-24T14:19:56.769 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:56.403+0000 7fc17254e700 1 -- 172.21.15.19:0/1072425117 wait complete. 2024-01-24T14:19:56.769 INFO:teuthology.orchestra.run.smithi019.stdout:Waiting for the mgr to restart... 2024-01-24T14:19:56.770 INFO:teuthology.orchestra.run.smithi019.stdout:Waiting for mgr epoch 9... 2024-01-24T14:19:56.978 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:56 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:56.887+0000 7f99dc943200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-24T14:19:57.235 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:57 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:57.163+0000 7f99dc943200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-24T14:19:57.567 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:19:57 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/1072425117' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-01-24T14:19:57.567 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:57 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:57.262+0000 7f99dc943200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-24T14:19:57.567 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:57 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:57.385+0000 7f99dc943200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-24T14:19:57.985 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:57 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:57.567+0000 7f99dc943200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-24T14:19:57.986 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:57 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:57.711+0000 7f99dc943200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-24T14:19:57.986 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:57 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:57.807+0000 7f99dc943200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-24T14:19:58.485 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:58 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:58.229+0000 7f99dc943200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-24T14:19:58.851 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:58 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:58.528+0000 7f99dc943200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-24T14:19:59.235 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:58 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:58.851+0000 7f99dc943200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-24T14:19:59.235 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:58 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:58.957+0000 7f99dc943200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-24T14:20:00.090 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:59 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:59.815+0000 7f99dc943200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-24T14:20:00.090 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:19:59 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:19:59.922+0000 7f99dc943200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-24T14:20:00.485 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:20:00 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:20:00.089+0000 7f99dc943200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-24T14:20:01.235 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:20:00 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:20:00.808+0000 7f99dc943200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-24T14:20:01.235 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:20:01 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:20:01.015+0000 7f99dc943200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-24T14:20:01.485 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:20:01 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:20:01.234+0000 7f99dc943200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-24T14:20:02.582 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:20:02 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:20:02.278+0000 7f99dc943200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-24T14:20:02.897 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:20:02 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:20:02.582+0000 7f99dc943200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-24T14:20:02.898 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:20:02 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:20:02.680+0000 7f99dc943200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-24T14:20:03.235 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:20:02 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:20:02.897+0000 7f99dc943200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-24T14:20:03.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:02 smithi019 ceph-mon[40663]: Active manager daemon x restarted 2024-01-24T14:20:03.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:02 smithi019 ceph-mon[40663]: Activating manager daemon x 2024-01-24T14:20:03.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:02 smithi019 ceph-mon[40663]: osdmap e3: 0 total, 0 up, 0 in 2024-01-24T14:20:03.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:02 smithi019 ceph-mon[40663]: mgrmap e10: x(active, starting, since 0.00394081s) 2024-01-24T14:20:03.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:02 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-01-24T14:20:03.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:02 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-01-24T14:20:03.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:02 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-24T14:20:03.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:02 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-24T14:20:03.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:02 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-24T14:20:03.237 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:02 smithi019 ceph-mon[40663]: Manager daemon x is now available 2024-01-24T14:20:03.237 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:02 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:20:03.237 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:02 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:20:04.235 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:03 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-01-24T14:20:04.235 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:03 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-01-24T14:20:04.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:03 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:04.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:03 smithi019 ceph-mon[40663]: mgrmap e11: x(active, since 1.00898s) 2024-01-24T14:20:04.284 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout { 2024-01-24T14:20:04.284 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-01-24T14:20:04.284 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout "initialized": true 2024-01-24T14:20:04.285 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout } 2024-01-24T14:20:04.285 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:57.237+0000 7f12bef0e700 1 Processor -- start 2024-01-24T14:20:04.285 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:57.237+0000 7f12bef0e700 1 -- start start 2024-01-24T14:20:04.285 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:57.237+0000 7f12bef0e700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f12b8107080 0x7f12b8109470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:04.285 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:57.237+0000 7f12bef0e700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b8079e30 con 0x7f12b8107080 2024-01-24T14:20:04.285 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:57.237+0000 7f12bccaa700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f12b8107080 0x7f12b8109470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:04.285 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:57.237+0000 7f12bccaa700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f12b8107080 0x7f12b8109470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:43700/0 (socket says 172.21.15.19:43700) 2024-01-24T14:20:04.285 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:57.237+0000 7f12bccaa700 1 -- 172.21.15.19:0/4194423111 learned_addr learned my addr 172.21.15.19:0/4194423111 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:20:04.285 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:57.238+0000 7f12bccaa700 1 -- 172.21.15.19:0/4194423111 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b81099b0 con 0x7f12b8107080 2024-01-24T14:20:04.285 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:57.238+0000 7f12bccaa700 1 --2- 172.21.15.19:0/4194423111 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f12b8107080 0x7f12b8109470 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f12ac0098a0 tx=0x7f12ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b4653953f1abbe1 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:20:04.285 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:57.238+0000 7f12b77fe700 1 -- 172.21.15.19:0/4194423111 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f12ac004160 con 0x7f12b8107080 2024-01-24T14:20:04.285 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:57.238+0000 7f12b77fe700 1 -- 172.21.15.19:0/4194423111 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f12ac0042c0 con 0x7f12b8107080 2024-01-24T14:20:04.285 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:57.238+0000 7f12b77fe700 1 -- 172.21.15.19:0/4194423111 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f12ac0119b0 con 0x7f12b8107080 2024-01-24T14:20:04.285 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:57.239+0000 7f12bef0e700 1 -- 172.21.15.19:0/4194423111 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f12b8107080 msgr2=0x7f12b8109470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:04.286 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:57.239+0000 7f12bef0e700 1 --2- 172.21.15.19:0/4194423111 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f12b8107080 0x7f12b8109470 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f12ac0098a0 tx=0x7f12ac009bb0 comp rx=0 tx=0).stop 2024-01-24T14:20:04.286 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:57.239+0000 7f12bef0e700 1 -- 172.21.15.19:0/4194423111 shutdown_connections 2024-01-24T14:20:04.286 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:57.239+0000 7f12bef0e700 1 --2- 172.21.15.19:0/4194423111 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f12b8107080 0x7f12b8109470 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:04.286 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:57.239+0000 7f12bef0e700 1 -- 172.21.15.19:0/4194423111 >> 172.21.15.19:0/4194423111 conn(0x7f12b8100a70 msgr2=0x7f12b8102e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:20:04.286 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:57.239+0000 7f12bef0e700 1 -- 172.21.15.19:0/4194423111 shutdown_connections 2024-01-24T14:20:04.286 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:57.239+0000 7f12bef0e700 1 -- 172.21.15.19:0/4194423111 wait complete. 2024-01-24T14:20:04.286 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:57.240+0000 7f12bef0e700 1 Processor -- start 2024-01-24T14:20:04.286 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:57.240+0000 7f12bef0e700 1 -- start start 2024-01-24T14:20:04.286 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:57.240+0000 7f12bef0e700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f12b8107080 0x7f12b819c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:04.286 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:57.240+0000 7f12bef0e700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b819c930 con 0x7f12b8107080 2024-01-24T14:20:04.286 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:57.240+0000 7f12bccaa700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f12b8107080 0x7f12b819c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:04.287 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:57.240+0000 7f12bccaa700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f12b8107080 0x7f12b819c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:43710/0 (socket says 172.21.15.19:43710) 2024-01-24T14:20:04.287 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:57.240+0000 7f12bccaa700 1 -- 172.21.15.19:0/4171771861 learned_addr learned my addr 172.21.15.19:0/4171771861 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:20:04.287 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:57.241+0000 7f12bccaa700 1 -- 172.21.15.19:0/4171771861 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12ac009550 con 0x7f12b8107080 2024-01-24T14:20:04.287 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:57.241+0000 7f12bccaa700 1 --2- 172.21.15.19:0/4171771861 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f12b8107080 0x7f12b819c3f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f12ac006af0 tx=0x7f12ac011a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:20:04.287 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:57.241+0000 7f12b5ffb700 1 -- 172.21.15.19:0/4171771861 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f12ac003a40 con 0x7f12b8107080 2024-01-24T14:20:04.287 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:57.241+0000 7f12b5ffb700 1 -- 172.21.15.19:0/4171771861 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f12ac003ba0 con 0x7f12b8107080 2024-01-24T14:20:04.287 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:57.241+0000 7f12bef0e700 1 -- 172.21.15.19:0/4171771861 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12b819cb30 con 0x7f12b8107080 2024-01-24T14:20:04.287 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:57.241+0000 7f12b5ffb700 1 -- 172.21.15.19:0/4171771861 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f12ac0196c0 con 0x7f12b8107080 2024-01-24T14:20:04.287 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:57.241+0000 7f12bef0e700 1 -- 172.21.15.19:0/4171771861 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12b819cf50 con 0x7f12b8107080 2024-01-24T14:20:04.287 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:57.242+0000 7f12b5ffb700 1 -- 172.21.15.19:0/4171771861 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 9) v1 ==== 45281+0+0 (secure 0 0 0) 0x7f12ac019980 con 0x7f12b8107080 2024-01-24T14:20:04.287 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:57.242+0000 7f12b5ffb700 1 --2- 172.21.15.19:0/4171771861 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7f12a80383c0 0x7f12a803a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:04.287 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:57.242+0000 7f12b5ffb700 1 -- 172.21.15.19:0/4171771861 --> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f12a803af90 con 0x7f12a80383c0 2024-01-24T14:20:04.288 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:57.242+0000 7f12b7fff700 1 -- 172.21.15.19:0/4171771861 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7f12a80383c0 msgr2=0x7f12a803a880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:6800/3758061162 2024-01-24T14:20:04.288 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:57.242+0000 7f12b5ffb700 1 -- 172.21.15.19:0/4171771861 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f12ac04c240 con 0x7f12b8107080 2024-01-24T14:20:04.288 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:57.243+0000 7f12b7fff700 1 --2- 172.21.15.19:0/4171771861 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7f12a80383c0 0x7f12a803a880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-24T14:20:04.288 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:57.444+0000 7f12b7fff700 1 -- 172.21.15.19:0/4171771861 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7f12a80383c0 msgr2=0x7f12a803a880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:6800/3758061162 2024-01-24T14:20:04.288 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:57.444+0000 7f12b7fff700 1 --2- 172.21.15.19:0/4171771861 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7f12a80383c0 0x7f12a803a880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-01-24T14:20:04.288 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:57.845+0000 7f12b7fff700 1 -- 172.21.15.19:0/4171771861 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7f12a80383c0 msgr2=0x7f12a803a880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:6800/3758061162 2024-01-24T14:20:04.288 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:57.845+0000 7f12b7fff700 1 --2- 172.21.15.19:0/4171771861 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7f12a80383c0 0x7f12a803a880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-01-24T14:20:04.288 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:58.646+0000 7f12b7fff700 1 -- 172.21.15.19:0/4171771861 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7f12a80383c0 msgr2=0x7f12a803a880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:6800/3758061162 2024-01-24T14:20:04.288 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:19:58.646+0000 7f12b7fff700 1 --2- 172.21.15.19:0/4171771861 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7f12a80383c0 0x7f12a803a880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-01-24T14:20:04.288 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:00.247+0000 7f12b7fff700 1 -- 172.21.15.19:0/4171771861 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7f12a80383c0 msgr2=0x7f12a803a880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.19:6800/3758061162 2024-01-24T14:20:04.288 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:00.247+0000 7f12b7fff700 1 --2- 172.21.15.19:0/4171771861 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7f12a80383c0 0x7f12a803a880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-01-24T14:20:04.288 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:02.913+0000 7f12b5ffb700 1 -- 172.21.15.19:0/4171771861 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mgrmap(e 10) v1 ==== 45048+0+0 (secure 0 0 0) 0x7f12ac00ed40 con 0x7f12b8107080 2024-01-24T14:20:04.289 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:02.913+0000 7f12b5ffb700 1 -- 172.21.15.19:0/4171771861 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7f12a80383c0 msgr2=0x7f12a803a880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-24T14:20:04.289 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:02.913+0000 7f12b5ffb700 1 --2- 172.21.15.19:0/4171771861 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7f12a80383c0 0x7f12a803a880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:04.289 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:03.917+0000 7f12b5ffb700 1 -- 172.21.15.19:0/4171771861 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mgrmap(e 11) v1 ==== 45175+0+0 (secure 0 0 0) 0x7f12ac02dcd0 con 0x7f12b8107080 2024-01-24T14:20:04.289 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:03.917+0000 7f12b5ffb700 1 --2- 172.21.15.19:0/4171771861 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f12a803be30 0x7f12a803e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:04.289 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:03.917+0000 7f12b5ffb700 1 -- 172.21.15.19:0/4171771861 --> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f12a803af90 con 0x7f12a803be30 2024-01-24T14:20:04.289 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:03.918+0000 7f12b7fff700 1 --2- 172.21.15.19:0/4171771861 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f12a803be30 0x7f12a803e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:04.289 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:03.919+0000 7f12b7fff700 1 --2- 172.21.15.19:0/4171771861 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f12a803be30 0x7f12a803e220 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f12a4003a10 tx=0x7f12a40092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:20:04.289 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:03.920+0000 7f12b5ffb700 1 -- 172.21.15.19:0/4171771861 <== mgr.14152 v2:172.21.15.19:6800/1634350318 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f12a803af90 con 0x7f12a803be30 2024-01-24T14:20:04.289 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:03.930+0000 7f12bef0e700 1 -- 172.21.15.19:0/4171771861 --> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f12b819d730 con 0x7f12a803be30 2024-01-24T14:20:04.289 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:03.930+0000 7f12b5ffb700 1 -- 172.21.15.19:0/4171771861 <== mgr.14152 v2:172.21.15.19:6800/1634350318 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f12b819d730 con 0x7f12a803be30 2024-01-24T14:20:04.289 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:03.931+0000 7f12bef0e700 1 -- 172.21.15.19:0/4171771861 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f12a803be30 msgr2=0x7f12a803e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:04.289 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:03.931+0000 7f12bef0e700 1 --2- 172.21.15.19:0/4171771861 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f12a803be30 0x7f12a803e220 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f12a4003a10 tx=0x7f12a40092b0 comp rx=0 tx=0).stop 2024-01-24T14:20:04.289 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:03.931+0000 7f12bef0e700 1 -- 172.21.15.19:0/4171771861 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f12b8107080 msgr2=0x7f12b819c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:04.289 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:03.931+0000 7f12bef0e700 1 --2- 172.21.15.19:0/4171771861 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f12b8107080 0x7f12b819c3f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f12ac006af0 tx=0x7f12ac011a10 comp rx=0 tx=0).stop 2024-01-24T14:20:04.290 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:03.931+0000 7f12bef0e700 1 -- 172.21.15.19:0/4171771861 shutdown_connections 2024-01-24T14:20:04.290 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:03.931+0000 7f12bef0e700 1 --2- 172.21.15.19:0/4171771861 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f12a803be30 0x7f12a803e220 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:04.290 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:03.931+0000 7f12bef0e700 1 --2- 172.21.15.19:0/4171771861 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f12b8107080 0x7f12b819c3f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:04.290 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:03.931+0000 7f12bef0e700 1 --2- 172.21.15.19:0/4171771861 >> [v2:172.21.15.19:6800/3758061162,v1:172.21.15.19:6801/3758061162] conn(0x7f12a80383c0 0x7f12a803a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:04.290 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:03.931+0000 7f12bef0e700 1 -- 172.21.15.19:0/4171771861 >> 172.21.15.19:0/4171771861 conn(0x7f12b8100a70 msgr2=0x7f12b81017d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:20:04.290 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:03.931+0000 7f12bef0e700 1 -- 172.21.15.19:0/4171771861 shutdown_connections 2024-01-24T14:20:04.290 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:03.931+0000 7f12bef0e700 1 -- 172.21.15.19:0/4171771861 wait complete. 2024-01-24T14:20:04.290 INFO:teuthology.orchestra.run.smithi019.stdout:mgr epoch 9 is available 2024-01-24T14:20:04.290 INFO:teuthology.orchestra.run.smithi019.stdout:Generating a dashboard self-signed certificate... 2024-01-24T14:20:05.412 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:05 smithi019 ceph-mon[40663]: [24/Jan/2024:14:20:03] ENGINE Bus STARTING 2024-01-24T14:20:05.412 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:05 smithi019 ceph-mon[40663]: [24/Jan/2024:14:20:03] ENGINE Serving on https://172.21.15.19:7150 2024-01-24T14:20:05.412 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:05 smithi019 ceph-mon[40663]: [24/Jan/2024:14:20:03] ENGINE Serving on http://172.21.15.19:8765 2024-01-24T14:20:05.412 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:05 smithi019 ceph-mon[40663]: [24/Jan/2024:14:20:03] ENGINE Bus STARTED 2024-01-24T14:20:05.412 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:05 smithi019 ceph-mon[40663]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-01-24T14:20:05.412 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:05 smithi019 ceph-mon[40663]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-01-24T14:20:05.412 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:05 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:05.413 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:05 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:05.413 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:05 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:05.506 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-01-24T14:20:05.506 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:04.671+0000 7f5ef53b9700 1 Processor -- start 2024-01-24T14:20:05.506 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:04.671+0000 7f5ef53b9700 1 -- start start 2024-01-24T14:20:05.506 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:04.671+0000 7f5ef53b9700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5ef0104860 0x7f5ef0104c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:05.506 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:04.671+0000 7f5ef53b9700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef0105250 con 0x7f5ef0104860 2024-01-24T14:20:05.507 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:04.671+0000 7f5eeeffd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5ef0104860 0x7f5ef0104c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:05.507 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:04.671+0000 7f5eeeffd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5ef0104860 0x7f5ef0104c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:35660/0 (socket says 172.21.15.19:35660) 2024-01-24T14:20:05.507 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:04.671+0000 7f5eeeffd700 1 -- 172.21.15.19:0/4126541337 learned_addr learned my addr 172.21.15.19:0/4126541337 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:20:05.507 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:04.672+0000 7f5eeeffd700 1 -- 172.21.15.19:0/4126541337 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef01077f0 con 0x7f5ef0104860 2024-01-24T14:20:05.507 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:04.672+0000 7f5eeeffd700 1 --2- 172.21.15.19:0/4126541337 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5ef0104860 0x7f5ef0104c80 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5ee00098a0 tx=0x7f5ee0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c70becc2a3b9481 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:20:05.507 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:04.672+0000 7f5eedffb700 1 -- 172.21.15.19:0/4126541337 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ee0004160 con 0x7f5ef0104860 2024-01-24T14:20:05.507 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:04.672+0000 7f5eedffb700 1 -- 172.21.15.19:0/4126541337 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5ee00042c0 con 0x7f5ef0104860 2024-01-24T14:20:05.507 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:04.672+0000 7f5eedffb700 1 -- 172.21.15.19:0/4126541337 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ee00119b0 con 0x7f5ef0104860 2024-01-24T14:20:05.507 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:04.673+0000 7f5ef53b9700 1 -- 172.21.15.19:0/4126541337 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5ef0104860 msgr2=0x7f5ef0104c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:05.507 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:04.673+0000 7f5ef53b9700 1 --2- 172.21.15.19:0/4126541337 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5ef0104860 0x7f5ef0104c80 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5ee00098a0 tx=0x7f5ee0009bb0 comp rx=0 tx=0).stop 2024-01-24T14:20:05.507 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:04.673+0000 7f5ef53b9700 1 -- 172.21.15.19:0/4126541337 shutdown_connections 2024-01-24T14:20:05.507 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:04.673+0000 7f5ef53b9700 1 --2- 172.21.15.19:0/4126541337 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5ef0104860 0x7f5ef0104c80 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:05.507 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:04.673+0000 7f5ef53b9700 1 -- 172.21.15.19:0/4126541337 >> 172.21.15.19:0/4126541337 conn(0x7f5ef0100190 msgr2=0x7f5ef01025d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:20:05.508 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:04.673+0000 7f5ef53b9700 1 -- 172.21.15.19:0/4126541337 shutdown_connections 2024-01-24T14:20:05.508 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:04.673+0000 7f5ef53b9700 1 -- 172.21.15.19:0/4126541337 wait complete. 2024-01-24T14:20:05.508 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:04.674+0000 7f5ef53b9700 1 Processor -- start 2024-01-24T14:20:05.508 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:04.674+0000 7f5ef53b9700 1 -- start start 2024-01-24T14:20:05.508 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:04.675+0000 7f5ef53b9700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5ef0104860 0x7f5ef019c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:05.508 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:04.675+0000 7f5ef53b9700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef019c880 con 0x7f5ef0104860 2024-01-24T14:20:05.508 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:04.675+0000 7f5eeeffd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5ef0104860 0x7f5ef019c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:05.508 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:04.675+0000 7f5eeeffd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5ef0104860 0x7f5ef019c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:35672/0 (socket says 172.21.15.19:35672) 2024-01-24T14:20:05.508 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:04.675+0000 7f5eeeffd700 1 -- 172.21.15.19:0/870424596 learned_addr learned my addr 172.21.15.19:0/870424596 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:20:05.508 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:04.675+0000 7f5eeeffd700 1 -- 172.21.15.19:0/870424596 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee0009550 con 0x7f5ef0104860 2024-01-24T14:20:05.508 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:04.675+0000 7f5eeeffd700 1 --2- 172.21.15.19:0/870424596 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5ef0104860 0x7f5ef019c340 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5ee0000c00 tx=0x7f5ee0011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:20:05.508 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:04.675+0000 7f5ed7fff700 1 -- 172.21.15.19:0/870424596 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ee00039f0 con 0x7f5ef0104860 2024-01-24T14:20:05.508 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:04.675+0000 7f5ef53b9700 1 -- 172.21.15.19:0/870424596 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ef019ca80 con 0x7f5ef0104860 2024-01-24T14:20:05.509 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:04.675+0000 7f5ed7fff700 1 -- 172.21.15.19:0/870424596 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5ee0003b50 con 0x7f5ef0104860 2024-01-24T14:20:05.509 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:04.675+0000 7f5ed7fff700 1 -- 172.21.15.19:0/870424596 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ee001a780 con 0x7f5ef0104860 2024-01-24T14:20:05.509 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:04.676+0000 7f5ef53b9700 1 -- 172.21.15.19:0/870424596 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ef019cea0 con 0x7f5ef0104860 2024-01-24T14:20:05.509 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:04.676+0000 7f5ed7fff700 1 -- 172.21.15.19:0/870424596 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 11) v1 ==== 45175+0+0 (secure 0 0 0) 0x7f5ee001a9f0 con 0x7f5ef0104860 2024-01-24T14:20:05.509 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:04.677+0000 7f5ed7fff700 1 --2- 172.21.15.19:0/870424596 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f5ed8038330 0x7f5ed803a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:05.509 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:04.677+0000 7f5ed7fff700 1 -- 172.21.15.19:0/870424596 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5ee004c1a0 con 0x7f5ef0104860 2024-01-24T14:20:05.509 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:04.677+0000 7f5ef53b9700 1 -- 172.21.15.19:0/870424596 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ef0062370 con 0x7f5ef0104860 2024-01-24T14:20:05.509 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:04.677+0000 7f5eee7fc700 1 --2- 172.21.15.19:0/870424596 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f5ed8038330 0x7f5ed803a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:05.509 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:04.677+0000 7f5eee7fc700 1 --2- 172.21.15.19:0/870424596 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f5ed8038330 0x7f5ed803a7f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5ee4006fd0 tx=0x7f5ee4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:20:05.509 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:04.682+0000 7f5ed7fff700 1 -- 172.21.15.19:0/870424596 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f5ee001acf0 con 0x7f5ef0104860 2024-01-24T14:20:05.509 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:04.899+0000 7f5ef53b9700 1 -- 172.21.15.19:0/870424596 --> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f5ef0101510 con 0x7f5ed8038330 2024-01-24T14:20:05.509 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:05.085+0000 7f5ed7fff700 1 -- 172.21.15.19:0/870424596 <== mgr.14152 v2:172.21.15.19:6800/1634350318 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f5ef0101510 con 0x7f5ed8038330 2024-01-24T14:20:05.509 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:05.087+0000 7f5ef53b9700 1 -- 172.21.15.19:0/870424596 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f5ed8038330 msgr2=0x7f5ed803a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:05.509 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:05.087+0000 7f5ef53b9700 1 --2- 172.21.15.19:0/870424596 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f5ed8038330 0x7f5ed803a7f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5ee4006fd0 tx=0x7f5ee4006e40 comp rx=0 tx=0).stop 2024-01-24T14:20:05.510 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:05.087+0000 7f5ef53b9700 1 -- 172.21.15.19:0/870424596 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5ef0104860 msgr2=0x7f5ef019c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:05.510 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:05.087+0000 7f5ef53b9700 1 --2- 172.21.15.19:0/870424596 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5ef0104860 0x7f5ef019c340 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5ee0000c00 tx=0x7f5ee0011a30 comp rx=0 tx=0).stop 2024-01-24T14:20:05.510 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:05.087+0000 7f5ef53b9700 1 -- 172.21.15.19:0/870424596 shutdown_connections 2024-01-24T14:20:05.510 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:05.087+0000 7f5ef53b9700 1 --2- 172.21.15.19:0/870424596 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f5ed8038330 0x7f5ed803a7f0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:05.510 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:05.087+0000 7f5ef53b9700 1 --2- 172.21.15.19:0/870424596 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5ef0104860 0x7f5ef019c340 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:05.510 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:05.087+0000 7f5ef53b9700 1 -- 172.21.15.19:0/870424596 >> 172.21.15.19:0/870424596 conn(0x7f5ef0100190 msgr2=0x7f5ef0100e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:20:05.510 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:05.087+0000 7f5ef53b9700 1 -- 172.21.15.19:0/870424596 shutdown_connections 2024-01-24T14:20:05.510 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:05.087+0000 7f5ef53b9700 1 -- 172.21.15.19:0/870424596 wait complete. 2024-01-24T14:20:05.510 INFO:teuthology.orchestra.run.smithi019.stdout:Creating initial admin user... 2024-01-24T14:20:06.673 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:06 smithi019 ceph-mon[40663]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:20:06.673 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:06 smithi019 ceph-mon[40663]: mgrmap e12: x(active, since 2s) 2024-01-24T14:20:06.880 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$HX.iL0Gxf6hXelNaBEI39uzjVXerNkY/MJZqnCKm5KwPdHGsD6av.", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1706106006, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-01-24T14:20:06.880 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.021+0000 7fa3e8f38700 1 Processor -- start 2024-01-24T14:20:06.881 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.021+0000 7fa3e8f38700 1 -- start start 2024-01-24T14:20:06.881 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.021+0000 7fa3e8f38700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa3e4104880 0x7fa3e4104ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:06.881 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.021+0000 7fa3e8f38700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e4105270 con 0x7fa3e4104880 2024-01-24T14:20:06.881 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.021+0000 7fa3e259c700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa3e4104880 0x7fa3e4104ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:06.881 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.021+0000 7fa3e259c700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa3e4104880 0x7fa3e4104ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:35680/0 (socket says 172.21.15.19:35680) 2024-01-24T14:20:06.881 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.021+0000 7fa3e259c700 1 -- 172.21.15.19:0/4155650880 learned_addr learned my addr 172.21.15.19:0/4155650880 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:20:06.881 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.022+0000 7fa3e259c700 1 -- 172.21.15.19:0/4155650880 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3e4107810 con 0x7fa3e4104880 2024-01-24T14:20:06.881 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.022+0000 7fa3e259c700 1 --2- 172.21.15.19:0/4155650880 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa3e4104880 0x7fa3e4104ca0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa3d40098a0 tx=0x7fa3d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d299ecf9a4c59522 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:20:06.881 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.022+0000 7fa3e159a700 1 -- 172.21.15.19:0/4155650880 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa3d4004160 con 0x7fa3e4104880 2024-01-24T14:20:06.881 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.022+0000 7fa3e159a700 1 -- 172.21.15.19:0/4155650880 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa3d40042c0 con 0x7fa3e4104880 2024-01-24T14:20:06.881 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.022+0000 7fa3e159a700 1 -- 172.21.15.19:0/4155650880 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa3d40119b0 con 0x7fa3e4104880 2024-01-24T14:20:06.881 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.022+0000 7fa3e8f38700 1 -- 172.21.15.19:0/4155650880 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa3e4104880 msgr2=0x7fa3e4104ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:06.881 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.022+0000 7fa3e8f38700 1 --2- 172.21.15.19:0/4155650880 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa3e4104880 0x7fa3e4104ca0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa3d40098a0 tx=0x7fa3d4009bb0 comp rx=0 tx=0).stop 2024-01-24T14:20:06.882 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.023+0000 7fa3e8f38700 1 -- 172.21.15.19:0/4155650880 shutdown_connections 2024-01-24T14:20:06.882 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.023+0000 7fa3e8f38700 1 --2- 172.21.15.19:0/4155650880 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa3e4104880 0x7fa3e4104ca0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:06.882 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.023+0000 7fa3e8f38700 1 -- 172.21.15.19:0/4155650880 >> 172.21.15.19:0/4155650880 conn(0x7fa3e41001f0 msgr2=0x7fa3e4102610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:20:06.882 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.023+0000 7fa3e8f38700 1 -- 172.21.15.19:0/4155650880 shutdown_connections 2024-01-24T14:20:06.882 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.023+0000 7fa3e8f38700 1 -- 172.21.15.19:0/4155650880 wait complete. 2024-01-24T14:20:06.882 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.024+0000 7fa3e8f38700 1 Processor -- start 2024-01-24T14:20:06.882 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.024+0000 7fa3e8f38700 1 -- start start 2024-01-24T14:20:06.882 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.024+0000 7fa3e8f38700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa3e4104880 0x7fa3e4107170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:06.882 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.024+0000 7fa3e8f38700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e41076b0 con 0x7fa3e4104880 2024-01-24T14:20:06.882 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.024+0000 7fa3e259c700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa3e4104880 0x7fa3e4107170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:06.882 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.024+0000 7fa3e259c700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa3e4104880 0x7fa3e4107170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:35692/0 (socket says 172.21.15.19:35692) 2024-01-24T14:20:06.882 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.024+0000 7fa3e259c700 1 -- 172.21.15.19:0/3867220377 learned_addr learned my addr 172.21.15.19:0/3867220377 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:20:06.882 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.024+0000 7fa3e259c700 1 -- 172.21.15.19:0/3867220377 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d4009550 con 0x7fa3e4104880 2024-01-24T14:20:06.882 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.024+0000 7fa3e259c700 1 --2- 172.21.15.19:0/3867220377 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa3e4104880 0x7fa3e4107170 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa3d4006af0 tx=0x7fa3d4011a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:20:06.883 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.025+0000 7fa3d37fe700 1 -- 172.21.15.19:0/3867220377 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa3d4003a40 con 0x7fa3e4104880 2024-01-24T14:20:06.883 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.025+0000 7fa3e8f38700 1 -- 172.21.15.19:0/3867220377 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3e41057c0 con 0x7fa3e4104880 2024-01-24T14:20:06.883 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.025+0000 7fa3d37fe700 1 -- 172.21.15.19:0/3867220377 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa3d4003ba0 con 0x7fa3e4104880 2024-01-24T14:20:06.883 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.025+0000 7fa3d37fe700 1 -- 172.21.15.19:0/3867220377 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa3d4019690 con 0x7fa3e4104880 2024-01-24T14:20:06.883 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.025+0000 7fa3e8f38700 1 -- 172.21.15.19:0/3867220377 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3e4105be0 con 0x7fa3e4104880 2024-01-24T14:20:06.883 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.026+0000 7fa3e8f38700 1 -- 172.21.15.19:0/3867220377 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3e410aa40 con 0x7fa3e4104880 2024-01-24T14:20:06.883 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.026+0000 7fa3d37fe700 1 -- 172.21.15.19:0/3867220377 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 12) v1 ==== 45281+0+0 (secure 0 0 0) 0x7fa3d401aa60 con 0x7fa3e4104880 2024-01-24T14:20:06.883 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.026+0000 7fa3d37fe700 1 --2- 172.21.15.19:0/3867220377 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fa3cc0383c0 0x7fa3cc03a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:06.883 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.026+0000 7fa3d37fe700 1 -- 172.21.15.19:0/3867220377 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa3d404c6c0 con 0x7fa3e4104880 2024-01-24T14:20:06.883 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.026+0000 7fa3e1d9b700 1 --2- 172.21.15.19:0/3867220377 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fa3cc0383c0 0x7fa3cc03a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:06.883 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.027+0000 7fa3e1d9b700 1 --2- 172.21.15.19:0/3867220377 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fa3cc0383c0 0x7fa3cc03a880 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa3d8006fd0 tx=0x7fa3d8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:20:06.883 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.031+0000 7fa3d37fe700 1 -- 172.21.15.19:0/3867220377 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fa3d4023750 con 0x7fa3e4104880 2024-01-24T14:20:06.883 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.252+0000 7fa3e8f38700 1 -- 172.21.15.19:0/3867220377 --> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fa3e4062190 con 0x7fa3cc0383c0 2024-01-24T14:20:06.884 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.508+0000 7fa3d37fe700 1 -- 172.21.15.19:0/3867220377 <== mgr.14152 v2:172.21.15.19:6800/1634350318 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7fa3e4062190 con 0x7fa3cc0383c0 2024-01-24T14:20:06.884 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.510+0000 7fa3e8f38700 1 -- 172.21.15.19:0/3867220377 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fa3cc0383c0 msgr2=0x7fa3cc03a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:06.884 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.510+0000 7fa3e8f38700 1 --2- 172.21.15.19:0/3867220377 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fa3cc0383c0 0x7fa3cc03a880 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa3d8006fd0 tx=0x7fa3d8006e40 comp rx=0 tx=0).stop 2024-01-24T14:20:06.884 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.510+0000 7fa3e8f38700 1 -- 172.21.15.19:0/3867220377 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa3e4104880 msgr2=0x7fa3e4107170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:06.884 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.510+0000 7fa3e8f38700 1 --2- 172.21.15.19:0/3867220377 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa3e4104880 0x7fa3e4107170 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa3d4006af0 tx=0x7fa3d4011a10 comp rx=0 tx=0).stop 2024-01-24T14:20:06.884 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.510+0000 7fa3e8f38700 1 -- 172.21.15.19:0/3867220377 shutdown_connections 2024-01-24T14:20:06.884 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.510+0000 7fa3e8f38700 1 --2- 172.21.15.19:0/3867220377 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fa3cc0383c0 0x7fa3cc03a880 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:06.884 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.510+0000 7fa3e8f38700 1 --2- 172.21.15.19:0/3867220377 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa3e4104880 0x7fa3e4107170 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:06.884 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.510+0000 7fa3e8f38700 1 -- 172.21.15.19:0/3867220377 >> 172.21.15.19:0/3867220377 conn(0x7fa3e41001f0 msgr2=0x7fa3e4100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:20:06.884 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.510+0000 7fa3e8f38700 1 -- 172.21.15.19:0/3867220377 shutdown_connections 2024-01-24T14:20:06.884 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:06.510+0000 7fa3e8f38700 1 -- 172.21.15.19:0/3867220377 wait complete. 2024-01-24T14:20:06.884 INFO:teuthology.orchestra.run.smithi019.stdout:Fetching dashboard port number... 2024-01-24T14:20:07.648 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:07 smithi019 ceph-mon[40663]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:20:07.648 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:07 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:07.831 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stdout 8443 2024-01-24T14:20:07.831 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.252+0000 7ff549960700 1 Processor -- start 2024-01-24T14:20:07.831 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.252+0000 7ff549960700 1 -- start start 2024-01-24T14:20:07.832 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.252+0000 7ff549960700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff544107030 0x7ff544109420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:07.832 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.252+0000 7ff549960700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff544079e70 con 0x7ff544107030 2024-01-24T14:20:07.832 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.252+0000 7ff542ffd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff544107030 0x7ff544109420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:07.832 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.252+0000 7ff542ffd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff544107030 0x7ff544109420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:35700/0 (socket says 172.21.15.19:35700) 2024-01-24T14:20:07.832 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.252+0000 7ff542ffd700 1 -- 172.21.15.19:0/3367583922 learned_addr learned my addr 172.21.15.19:0/3367583922 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:20:07.832 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.252+0000 7ff542ffd700 1 -- 172.21.15.19:0/3367583922 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff544109960 con 0x7ff544107030 2024-01-24T14:20:07.832 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.253+0000 7ff542ffd700 1 --2- 172.21.15.19:0/3367583922 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff544107030 0x7ff544109420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff5340098a0 tx=0x7ff534009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d848d327c94eaa server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:20:07.833 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.253+0000 7ff5427fc700 1 -- 172.21.15.19:0/3367583922 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff534004160 con 0x7ff544107030 2024-01-24T14:20:07.833 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.253+0000 7ff5427fc700 1 -- 172.21.15.19:0/3367583922 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff5340042c0 con 0x7ff544107030 2024-01-24T14:20:07.833 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.253+0000 7ff5427fc700 1 -- 172.21.15.19:0/3367583922 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff5340119b0 con 0x7ff544107030 2024-01-24T14:20:07.833 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.253+0000 7ff549960700 1 -- 172.21.15.19:0/3367583922 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff544107030 msgr2=0x7ff544109420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:07.833 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.253+0000 7ff549960700 1 --2- 172.21.15.19:0/3367583922 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff544107030 0x7ff544109420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff5340098a0 tx=0x7ff534009bb0 comp rx=0 tx=0).stop 2024-01-24T14:20:07.833 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.254+0000 7ff549960700 1 -- 172.21.15.19:0/3367583922 shutdown_connections 2024-01-24T14:20:07.834 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.254+0000 7ff549960700 1 --2- 172.21.15.19:0/3367583922 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff544107030 0x7ff544109420 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:07.834 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.254+0000 7ff549960700 1 -- 172.21.15.19:0/3367583922 >> 172.21.15.19:0/3367583922 conn(0x7ff544100a40 msgr2=0x7ff544102e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:20:07.834 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.254+0000 7ff549960700 1 -- 172.21.15.19:0/3367583922 shutdown_connections 2024-01-24T14:20:07.834 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.254+0000 7ff549960700 1 -- 172.21.15.19:0/3367583922 wait complete. 2024-01-24T14:20:07.834 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.254+0000 7ff549960700 1 Processor -- start 2024-01-24T14:20:07.834 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.254+0000 7ff549960700 1 -- start start 2024-01-24T14:20:07.834 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.255+0000 7ff549960700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff544107030 0x7ff54419c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:07.834 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.255+0000 7ff549960700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff54419c930 con 0x7ff544107030 2024-01-24T14:20:07.834 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.255+0000 7ff542ffd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff544107030 0x7ff54419c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:07.834 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.255+0000 7ff542ffd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff544107030 0x7ff54419c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:35704/0 (socket says 172.21.15.19:35704) 2024-01-24T14:20:07.835 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.255+0000 7ff542ffd700 1 -- 172.21.15.19:0/241347937 learned_addr learned my addr 172.21.15.19:0/241347937 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:20:07.835 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.255+0000 7ff542ffd700 1 -- 172.21.15.19:0/241347937 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff534009550 con 0x7ff544107030 2024-01-24T14:20:07.835 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.255+0000 7ff542ffd700 1 --2- 172.21.15.19:0/241347937 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff544107030 0x7ff54419c3f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff534006af0 tx=0x7ff534011a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:20:07.835 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.255+0000 7ff54895e700 1 -- 172.21.15.19:0/241347937 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff534003a40 con 0x7ff544107030 2024-01-24T14:20:07.835 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.255+0000 7ff54895e700 1 -- 172.21.15.19:0/241347937 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff534003ba0 con 0x7ff544107030 2024-01-24T14:20:07.835 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.255+0000 7ff549960700 1 -- 172.21.15.19:0/241347937 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff54419cb30 con 0x7ff544107030 2024-01-24T14:20:07.836 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.256+0000 7ff54895e700 1 -- 172.21.15.19:0/241347937 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff5340196c0 con 0x7ff544107030 2024-01-24T14:20:07.836 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.256+0000 7ff549960700 1 -- 172.21.15.19:0/241347937 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff54419cf50 con 0x7ff544107030 2024-01-24T14:20:07.836 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.257+0000 7ff549960700 1 -- 172.21.15.19:0/241347937 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff544062370 con 0x7ff544107030 2024-01-24T14:20:07.836 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.257+0000 7ff54895e700 1 -- 172.21.15.19:0/241347937 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 12) v1 ==== 45281+0+0 (secure 0 0 0) 0x7ff53401aa80 con 0x7ff544107030 2024-01-24T14:20:07.836 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.257+0000 7ff54895e700 1 --2- 172.21.15.19:0/241347937 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7ff5300383c0 0x7ff53003a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:07.836 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.257+0000 7ff54895e700 1 -- 172.21.15.19:0/241347937 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff53404cd60 con 0x7ff544107030 2024-01-24T14:20:07.836 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.257+0000 7ff53a7fc700 1 --2- 172.21.15.19:0/241347937 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7ff5300383c0 0x7ff53003a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:07.836 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.258+0000 7ff53a7fc700 1 --2- 172.21.15.19:0/241347937 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7ff5300383c0 0x7ff53003a880 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff52c006fd0 tx=0x7ff52c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:20:07.837 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.262+0000 7ff54895e700 1 -- 172.21.15.19:0/241347937 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7ff534023750 con 0x7ff544107030 2024-01-24T14:20:07.837 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.468+0000 7ff549960700 1 -- 172.21.15.19:0/241347937 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7ff54419fdb0 con 0x7ff544107030 2024-01-24T14:20:07.837 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.469+0000 7ff54895e700 1 -- 172.21.15.19:0/241347937 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7ff53402c3e0 con 0x7ff544107030 2024-01-24T14:20:07.837 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.471+0000 7ff549960700 1 -- 172.21.15.19:0/241347937 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7ff5300383c0 msgr2=0x7ff53003a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:07.837 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.471+0000 7ff549960700 1 --2- 172.21.15.19:0/241347937 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7ff5300383c0 0x7ff53003a880 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff52c006fd0 tx=0x7ff52c006e40 comp rx=0 tx=0).stop 2024-01-24T14:20:07.837 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.471+0000 7ff549960700 1 -- 172.21.15.19:0/241347937 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff544107030 msgr2=0x7ff54419c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:07.837 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.471+0000 7ff549960700 1 --2- 172.21.15.19:0/241347937 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff544107030 0x7ff54419c3f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff534006af0 tx=0x7ff534011a10 comp rx=0 tx=0).stop 2024-01-24T14:20:07.837 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.471+0000 7ff549960700 1 -- 172.21.15.19:0/241347937 shutdown_connections 2024-01-24T14:20:07.837 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.471+0000 7ff549960700 1 --2- 172.21.15.19:0/241347937 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7ff5300383c0 0x7ff53003a880 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:07.837 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.471+0000 7ff549960700 1 --2- 172.21.15.19:0/241347937 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff544107030 0x7ff54419c3f0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:07.837 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.471+0000 7ff549960700 1 -- 172.21.15.19:0/241347937 >> 172.21.15.19:0/241347937 conn(0x7ff544100a40 msgr2=0x7ff544101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:20:07.838 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.471+0000 7ff549960700 1 -- 172.21.15.19:0/241347937 shutdown_connections 2024-01-24T14:20:07.838 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:07.471+0000 7ff549960700 1 -- 172.21.15.19:0/241347937 wait complete. 2024-01-24T14:20:07.841 INFO:teuthology.orchestra.run.smithi019.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-01-24T14:20:07.841 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stdout disabled 2024-01-24T14:20:07.850 INFO:teuthology.orchestra.run.smithi019.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-01-24T14:20:07.850 INFO:teuthology.orchestra.run.smithi019.stdout:systemctl: stdout inactive 2024-01-24T14:20:07.850 INFO:teuthology.orchestra.run.smithi019.stdout:firewalld.service is not enabled 2024-01-24T14:20:07.851 INFO:teuthology.orchestra.run.smithi019.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-01-24T14:20:07.852 INFO:teuthology.orchestra.run.smithi019.stdout:Ceph Dashboard is now available at: 2024-01-24T14:20:07.852 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:20:07.852 INFO:teuthology.orchestra.run.smithi019.stdout: URL: https://smithi019:8443/ 2024-01-24T14:20:07.852 INFO:teuthology.orchestra.run.smithi019.stdout: User: admin 2024-01-24T14:20:07.852 INFO:teuthology.orchestra.run.smithi019.stdout: Password: 5r1qrtmro6 2024-01-24T14:20:07.852 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:20:07.853 INFO:teuthology.orchestra.run.smithi019.stdout:Saving cluster configuration to /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/config directory 2024-01-24T14:20:07.853 INFO:teuthology.orchestra.run.smithi019.stdout:Enabling autotune for osd_memory_target 2024-01-24T14:20:08.711 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:08 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/241347937' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-01-24T14:20:08.814 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.239+0000 7fd03553f700 1 Processor -- start 2024-01-24T14:20:08.814 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.239+0000 7fd03553f700 1 -- start start 2024-01-24T14:20:08.814 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.239+0000 7fd03553f700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd030104860 0x7fd030104c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:08.814 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.239+0000 7fd03553f700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd030105250 con 0x7fd030104860 2024-01-24T14:20:08.814 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.239+0000 7fd02effd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd030104860 0x7fd030104c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:08.814 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.239+0000 7fd02effd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd030104860 0x7fd030104c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:35708/0 (socket says 172.21.15.19:35708) 2024-01-24T14:20:08.814 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.239+0000 7fd02effd700 1 -- 172.21.15.19:0/1207519490 learned_addr learned my addr 172.21.15.19:0/1207519490 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:20:08.814 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.239+0000 7fd02effd700 1 -- 172.21.15.19:0/1207519490 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0301077f0 con 0x7fd030104860 2024-01-24T14:20:08.814 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.240+0000 7fd02effd700 1 --2- 172.21.15.19:0/1207519490 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd030104860 0x7fd030104c80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd0200098a0 tx=0x7fd020009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e917be54875a88 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:20:08.814 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.240+0000 7fd02dffb700 1 -- 172.21.15.19:0/1207519490 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd020004160 con 0x7fd030104860 2024-01-24T14:20:08.815 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.240+0000 7fd02dffb700 1 -- 172.21.15.19:0/1207519490 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd0200042c0 con 0x7fd030104860 2024-01-24T14:20:08.815 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.240+0000 7fd02dffb700 1 -- 172.21.15.19:0/1207519490 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd0200119b0 con 0x7fd030104860 2024-01-24T14:20:08.815 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.240+0000 7fd03553f700 1 -- 172.21.15.19:0/1207519490 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd030104860 msgr2=0x7fd030104c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:08.815 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.240+0000 7fd03553f700 1 --2- 172.21.15.19:0/1207519490 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd030104860 0x7fd030104c80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd0200098a0 tx=0x7fd020009bb0 comp rx=0 tx=0).stop 2024-01-24T14:20:08.815 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.240+0000 7fd03553f700 1 -- 172.21.15.19:0/1207519490 shutdown_connections 2024-01-24T14:20:08.815 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.240+0000 7fd03553f700 1 --2- 172.21.15.19:0/1207519490 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd030104860 0x7fd030104c80 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:08.815 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.240+0000 7fd03553f700 1 -- 172.21.15.19:0/1207519490 >> 172.21.15.19:0/1207519490 conn(0x7fd030100190 msgr2=0x7fd0301025d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:20:08.815 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.241+0000 7fd03553f700 1 -- 172.21.15.19:0/1207519490 shutdown_connections 2024-01-24T14:20:08.815 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.241+0000 7fd03553f700 1 -- 172.21.15.19:0/1207519490 wait complete. 2024-01-24T14:20:08.815 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.241+0000 7fd03553f700 1 Processor -- start 2024-01-24T14:20:08.815 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.241+0000 7fd03553f700 1 -- start start 2024-01-24T14:20:08.815 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.242+0000 7fd03553f700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd030104860 0x7fd03019a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:08.815 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.242+0000 7fd03553f700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd03019a680 con 0x7fd030104860 2024-01-24T14:20:08.816 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.242+0000 7fd02effd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd030104860 0x7fd03019a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:08.816 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.242+0000 7fd02effd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd030104860 0x7fd03019a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:35710/0 (socket says 172.21.15.19:35710) 2024-01-24T14:20:08.816 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.242+0000 7fd02effd700 1 -- 172.21.15.19:0/3837599578 learned_addr learned my addr 172.21.15.19:0/3837599578 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:20:08.816 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.242+0000 7fd02effd700 1 -- 172.21.15.19:0/3837599578 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd020009550 con 0x7fd030104860 2024-01-24T14:20:08.816 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.242+0000 7fd02effd700 1 --2- 172.21.15.19:0/3837599578 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd030104860 0x7fd03019a140 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd020003e60 tx=0x7fd020011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:20:08.816 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.242+0000 7fd017fff700 1 -- 172.21.15.19:0/3837599578 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd020003970 con 0x7fd030104860 2024-01-24T14:20:08.816 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.242+0000 7fd03553f700 1 -- 172.21.15.19:0/3837599578 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd03019a880 con 0x7fd030104860 2024-01-24T14:20:08.816 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.242+0000 7fd017fff700 1 -- 172.21.15.19:0/3837599578 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd020003ad0 con 0x7fd030104860 2024-01-24T14:20:08.816 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.242+0000 7fd017fff700 1 -- 172.21.15.19:0/3837599578 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd0200195e0 con 0x7fd030104860 2024-01-24T14:20:08.816 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.242+0000 7fd03553f700 1 -- 172.21.15.19:0/3837599578 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd03019aca0 con 0x7fd030104860 2024-01-24T14:20:08.816 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.243+0000 7fd017fff700 1 -- 172.21.15.19:0/3837599578 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 12) v1 ==== 45281+0+0 (secure 0 0 0) 0x7fd0200198a0 con 0x7fd030104860 2024-01-24T14:20:08.816 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.243+0000 7fd03553f700 1 -- 172.21.15.19:0/3837599578 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd030062370 con 0x7fd030104860 2024-01-24T14:20:08.816 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.243+0000 7fd017fff700 1 --2- 172.21.15.19:0/3837599578 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fd0180383c0 0x7fd01803a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:08.817 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.244+0000 7fd017fff700 1 -- 172.21.15.19:0/3837599578 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd02004c4a0 con 0x7fd030104860 2024-01-24T14:20:08.817 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.244+0000 7fd02e7fc700 1 --2- 172.21.15.19:0/3837599578 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fd0180383c0 0x7fd01803a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:08.817 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.244+0000 7fd02e7fc700 1 --2- 172.21.15.19:0/3837599578 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fd0180383c0 0x7fd01803a880 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd024006fd0 tx=0x7fd024006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:20:08.817 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.248+0000 7fd017fff700 1 -- 172.21.15.19:0/3837599578 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fd02002b340 con 0x7fd030104860 2024-01-24T14:20:08.817 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.453+0000 7fd03553f700 1 -- 172.21.15.19:0/3837599578 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7fd03019db00 con 0x7fd030104860 2024-01-24T14:20:08.817 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.453+0000 7fd017fff700 1 -- 172.21.15.19:0/3837599578 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7fd020018b40 con 0x7fd030104860 2024-01-24T14:20:08.817 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.455+0000 7fd03553f700 1 -- 172.21.15.19:0/3837599578 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fd0180383c0 msgr2=0x7fd01803a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:08.817 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.455+0000 7fd03553f700 1 --2- 172.21.15.19:0/3837599578 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fd0180383c0 0x7fd01803a880 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd024006fd0 tx=0x7fd024006e40 comp rx=0 tx=0).stop 2024-01-24T14:20:08.817 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.455+0000 7fd03553f700 1 -- 172.21.15.19:0/3837599578 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd030104860 msgr2=0x7fd03019a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:08.817 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.455+0000 7fd03553f700 1 --2- 172.21.15.19:0/3837599578 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd030104860 0x7fd03019a140 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd020003e60 tx=0x7fd020011aa0 comp rx=0 tx=0).stop 2024-01-24T14:20:08.817 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.455+0000 7fd03553f700 1 -- 172.21.15.19:0/3837599578 shutdown_connections 2024-01-24T14:20:08.817 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.455+0000 7fd03553f700 1 --2- 172.21.15.19:0/3837599578 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fd0180383c0 0x7fd01803a880 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:08.817 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.455+0000 7fd03553f700 1 --2- 172.21.15.19:0/3837599578 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd030104860 0x7fd03019a140 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:08.818 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.455+0000 7fd03553f700 1 -- 172.21.15.19:0/3837599578 >> 172.21.15.19:0/3837599578 conn(0x7fd030100190 msgr2=0x7fd030100e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:20:08.818 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.456+0000 7fd03553f700 1 -- 172.21.15.19:0/3837599578 shutdown_connections 2024-01-24T14:20:08.818 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:08.456+0000 7fd03553f700 1 -- 172.21.15.19:0/3837599578 wait complete. 2024-01-24T14:20:09.871 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.172+0000 7f4840e40700 1 Processor -- start 2024-01-24T14:20:09.871 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.172+0000 7f4840e40700 1 -- start start 2024-01-24T14:20:09.871 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.173+0000 7f4840e40700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f483c106580 0x7f483c1069a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:09.871 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.173+0000 7f4840e40700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483c106f70 con 0x7f483c106580 2024-01-24T14:20:09.872 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.173+0000 7f483a59c700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f483c106580 0x7f483c1069a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:09.872 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.173+0000 7f483a59c700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f483c106580 0x7f483c1069a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:35724/0 (socket says 172.21.15.19:35724) 2024-01-24T14:20:09.872 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.173+0000 7f483a59c700 1 -- 172.21.15.19:0/4147490303 learned_addr learned my addr 172.21.15.19:0/4147490303 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:20:09.872 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.173+0000 7f483a59c700 1 -- 172.21.15.19:0/4147490303 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f483c107780 con 0x7f483c106580 2024-01-24T14:20:09.872 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.173+0000 7f483a59c700 1 --2- 172.21.15.19:0/4147490303 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f483c106580 0x7f483c1069a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f48240098a0 tx=0x7f4824009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ac065ff2fd1a4a3 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:20:09.872 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.173+0000 7f483959a700 1 -- 172.21.15.19:0/4147490303 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4824004160 con 0x7f483c106580 2024-01-24T14:20:09.872 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.174+0000 7f483959a700 1 -- 172.21.15.19:0/4147490303 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f48240042c0 con 0x7f483c106580 2024-01-24T14:20:09.872 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.174+0000 7f483959a700 1 -- 172.21.15.19:0/4147490303 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f48240119b0 con 0x7f483c106580 2024-01-24T14:20:09.872 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.174+0000 7f4840e40700 1 -- 172.21.15.19:0/4147490303 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f483c106580 msgr2=0x7f483c1069a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:09.872 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.174+0000 7f4840e40700 1 --2- 172.21.15.19:0/4147490303 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f483c106580 0x7f483c1069a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f48240098a0 tx=0x7f4824009bb0 comp rx=0 tx=0).stop 2024-01-24T14:20:09.872 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.174+0000 7f4840e40700 1 -- 172.21.15.19:0/4147490303 shutdown_connections 2024-01-24T14:20:09.872 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.174+0000 7f4840e40700 1 --2- 172.21.15.19:0/4147490303 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f483c106580 0x7f483c1069a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:09.872 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.174+0000 7f4840e40700 1 -- 172.21.15.19:0/4147490303 >> 172.21.15.19:0/4147490303 conn(0x7f483c101b60 msgr2=0x7f483c103f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:20:09.872 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.174+0000 7f4840e40700 1 -- 172.21.15.19:0/4147490303 shutdown_connections 2024-01-24T14:20:09.872 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.174+0000 7f4840e40700 1 -- 172.21.15.19:0/4147490303 wait complete. 2024-01-24T14:20:09.873 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.175+0000 7f4840e40700 1 Processor -- start 2024-01-24T14:20:09.873 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.175+0000 7f4840e40700 1 -- start start 2024-01-24T14:20:09.873 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.176+0000 7f4840e40700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f483c106580 0x7f483c19a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:09.873 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.176+0000 7f4840e40700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483c19a730 con 0x7f483c106580 2024-01-24T14:20:09.873 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.176+0000 7f483a59c700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f483c106580 0x7f483c19a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:09.873 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.176+0000 7f483a59c700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f483c106580 0x7f483c19a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:35730/0 (socket says 172.21.15.19:35730) 2024-01-24T14:20:09.873 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.176+0000 7f483a59c700 1 -- 172.21.15.19:0/355320621 learned_addr learned my addr 172.21.15.19:0/355320621 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:20:09.873 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.176+0000 7f483a59c700 1 -- 172.21.15.19:0/355320621 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4824009550 con 0x7f483c106580 2024-01-24T14:20:09.873 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.176+0000 7f483a59c700 1 --2- 172.21.15.19:0/355320621 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f483c106580 0x7f483c19a1f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4824006af0 tx=0x7f4824011a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:20:09.873 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.176+0000 7f48337fe700 1 -- 172.21.15.19:0/355320621 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4824003a40 con 0x7f483c106580 2024-01-24T14:20:09.873 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.176+0000 7f48337fe700 1 -- 172.21.15.19:0/355320621 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4824003ba0 con 0x7f483c106580 2024-01-24T14:20:09.873 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.176+0000 7f4840e40700 1 -- 172.21.15.19:0/355320621 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f483c19a930 con 0x7f483c106580 2024-01-24T14:20:09.873 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.176+0000 7f48337fe700 1 -- 172.21.15.19:0/355320621 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4824019690 con 0x7f483c106580 2024-01-24T14:20:09.873 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.177+0000 7f4840e40700 1 -- 172.21.15.19:0/355320621 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f483c19ad50 con 0x7f483c106580 2024-01-24T14:20:09.873 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.177+0000 7f48337fe700 1 -- 172.21.15.19:0/355320621 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 12) v1 ==== 45281+0+0 (secure 0 0 0) 0x7f4824019900 con 0x7f483c106580 2024-01-24T14:20:09.873 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.178+0000 7f48337fe700 1 --2- 172.21.15.19:0/355320621 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f4828038410 0x7f482803a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:09.874 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.178+0000 7f48337fe700 1 -- 172.21.15.19:0/355320621 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f482404c6f0 con 0x7f483c106580 2024-01-24T14:20:09.874 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.178+0000 7f4840e40700 1 -- 172.21.15.19:0/355320621 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f483c062370 con 0x7f483c106580 2024-01-24T14:20:09.874 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.178+0000 7f4839d9b700 1 --2- 172.21.15.19:0/355320621 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f4828038410 0x7f482803a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:09.874 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.178+0000 7f4839d9b700 1 --2- 172.21.15.19:0/355320621 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f4828038410 0x7f482803a8d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f482c006fd0 tx=0x7f482c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:20:09.874 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.183+0000 7f48337fe700 1 -- 172.21.15.19:0/355320621 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f48240243f0 con 0x7f483c106580 2024-01-24T14:20:09.874 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.363+0000 7f48337fe700 1 -- 172.21.15.19:0/355320621 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mgrmap(e 13) v1 ==== 45324+0+0 (secure 0 0 0) 0x7f482401aa60 con 0x7f483c106580 2024-01-24T14:20:09.874 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.506+0000 7f4840e40700 1 -- 172.21.15.19:0/355320621 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f483c19d790 con 0x7f483c106580 2024-01-24T14:20:09.874 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.508+0000 7f48337fe700 1 -- 172.21.15.19:0/355320621 <== mon.0 v2:172.21.15.19:3300/0 8 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7f48240243f0 con 0x7f483c106580 2024-01-24T14:20:09.874 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.511+0000 7f4840e40700 1 -- 172.21.15.19:0/355320621 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f4828038410 msgr2=0x7f482803a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:09.874 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.511+0000 7f4840e40700 1 --2- 172.21.15.19:0/355320621 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f4828038410 0x7f482803a8d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f482c006fd0 tx=0x7f482c006e40 comp rx=0 tx=0).stop 2024-01-24T14:20:09.874 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.511+0000 7f4840e40700 1 -- 172.21.15.19:0/355320621 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f483c106580 msgr2=0x7f483c19a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:09.874 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.511+0000 7f4840e40700 1 --2- 172.21.15.19:0/355320621 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f483c106580 0x7f483c19a1f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4824006af0 tx=0x7f4824011a10 comp rx=0 tx=0).stop 2024-01-24T14:20:09.874 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.511+0000 7f4840e40700 1 -- 172.21.15.19:0/355320621 shutdown_connections 2024-01-24T14:20:09.874 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.511+0000 7f4840e40700 1 --2- 172.21.15.19:0/355320621 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f4828038410 0x7f482803a8d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:09.874 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.511+0000 7f4840e40700 1 --2- 172.21.15.19:0/355320621 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f483c106580 0x7f483c19a1f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:09.875 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.511+0000 7f4840e40700 1 -- 172.21.15.19:0/355320621 >> 172.21.15.19:0/355320621 conn(0x7f483c101b60 msgr2=0x7f483c1028e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:20:09.875 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.511+0000 7f4840e40700 1 -- 172.21.15.19:0/355320621 shutdown_connections 2024-01-24T14:20:09.875 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr 2024-01-24T14:20:09.511+0000 7f4840e40700 1 -- 172.21.15.19:0/355320621 wait complete. 2024-01-24T14:20:09.875 INFO:teuthology.orchestra.run.smithi019.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-01-24T14:20:09.875 INFO:teuthology.orchestra.run.smithi019.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-01-24T14:20:09.875 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:20:09.875 INFO:teuthology.orchestra.run.smithi019.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-01-24T14:20:09.875 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:20:09.875 INFO:teuthology.orchestra.run.smithi019.stdout:Or, if you are only running a single cluster on this host: 2024-01-24T14:20:09.875 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:20:09.875 INFO:teuthology.orchestra.run.smithi019.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-01-24T14:20:09.875 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:20:09.875 INFO:teuthology.orchestra.run.smithi019.stdout:Please consider enabling telemetry to help improve Ceph: 2024-01-24T14:20:09.875 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:20:09.875 INFO:teuthology.orchestra.run.smithi019.stdout: ceph telemetry on 2024-01-24T14:20:09.875 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:20:09.876 INFO:teuthology.orchestra.run.smithi019.stdout:For more information see: 2024-01-24T14:20:09.876 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:20:09.876 INFO:teuthology.orchestra.run.smithi019.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-01-24T14:20:09.876 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:20:09.876 INFO:teuthology.orchestra.run.smithi019.stdout:Bootstrap complete. 2024-01-24T14:20:09.919 INFO:tasks.cephadm:Fetching config... 2024-01-24T14:20:09.919 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-01-24T14:20:09.920 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-01-24T14:20:09.939 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-01-24T14:20:09.939 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-01-24T14:20:09.940 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-01-24T14:20:10.000 INFO:tasks.cephadm:Fetching mon keyring... 2024-01-24T14:20:10.000 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-01-24T14:20:10.001 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/mon.a/keyring of=/dev/stdout 2024-01-24T14:20:10.079 INFO:tasks.cephadm:Fetching pub ssh key... 2024-01-24T14:20:10.079 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-01-24T14:20:10.079 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-01-24T14:20:10.140 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-01-24T14:20:10.140 DEBUG:teuthology.orchestra.run.smithi019:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRBhvYBWmdATrSMvfWo6Pk/4oJflz89pkUJtpO7iZnUX31bWEnSrytuzacicjMV3MQ+QQkzGDZKaklWSetQGxNzyFpiRBVx+xlHz46XKSOd+8kYijwC/k0EGzuarszGkvIyvA/3EuKKs/QHQVcns1V3nPT+STZ6vjtJt0BlL5+x7VvYfnjyuwSgFLv7ipoMwCsV4IFbrxc1WrfKdNfjHf+lC3dzfI5gokxCrS+ZlCmzEP4vFrqqFKvIEvkDXsnZNuvZI4drAEIrhdUNYOmoiqOZZ1mH5WAv4K243R/lAFQbsSKkYXDAoEnYfDikIjFYalQUl7+VwbHIqNLi800cECQnsF6XO18CYeoMD85URhRmIxYnsHW4Yi24KVQq0ah/2knyYJMDJVkXwsJArFwMgD1kmVaSjaA+zgKIbfYnkOg1p0bPEcFLcDiZnkPEKM7CWIJDimupfbp4zXs/iZ0B0HOgj7UZPsLn0RWjfPly7tNaDjw1DTpFRXLs+AWQVEMGD8= ceph-654b0c8e-bac3-11ee-95b1-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-24T14:20:10.224 INFO:teuthology.orchestra.run.smithi019.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRBhvYBWmdATrSMvfWo6Pk/4oJflz89pkUJtpO7iZnUX31bWEnSrytuzacicjMV3MQ+QQkzGDZKaklWSetQGxNzyFpiRBVx+xlHz46XKSOd+8kYijwC/k0EGzuarszGkvIyvA/3EuKKs/QHQVcns1V3nPT+STZ6vjtJt0BlL5+x7VvYfnjyuwSgFLv7ipoMwCsV4IFbrxc1WrfKdNfjHf+lC3dzfI5gokxCrS+ZlCmzEP4vFrqqFKvIEvkDXsnZNuvZI4drAEIrhdUNYOmoiqOZZ1mH5WAv4K243R/lAFQbsSKkYXDAoEnYfDikIjFYalQUl7+VwbHIqNLi800cECQnsF6XO18CYeoMD85URhRmIxYnsHW4Yi24KVQq0ah/2knyYJMDJVkXwsJArFwMgD1kmVaSjaA+zgKIbfYnkOg1p0bPEcFLcDiZnkPEKM7CWIJDimupfbp4zXs/iZ0B0HOgj7UZPsLn0RWjfPly7tNaDjw1DTpFRXLs+AWQVEMGD8= ceph-654b0c8e-bac3-11ee-95b1-87774f69a715 2024-01-24T14:20:10.240 DEBUG:teuthology.orchestra.run.smithi123:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRBhvYBWmdATrSMvfWo6Pk/4oJflz89pkUJtpO7iZnUX31bWEnSrytuzacicjMV3MQ+QQkzGDZKaklWSetQGxNzyFpiRBVx+xlHz46XKSOd+8kYijwC/k0EGzuarszGkvIyvA/3EuKKs/QHQVcns1V3nPT+STZ6vjtJt0BlL5+x7VvYfnjyuwSgFLv7ipoMwCsV4IFbrxc1WrfKdNfjHf+lC3dzfI5gokxCrS+ZlCmzEP4vFrqqFKvIEvkDXsnZNuvZI4drAEIrhdUNYOmoiqOZZ1mH5WAv4K243R/lAFQbsSKkYXDAoEnYfDikIjFYalQUl7+VwbHIqNLi800cECQnsF6XO18CYeoMD85URhRmIxYnsHW4Yi24KVQq0ah/2knyYJMDJVkXwsJArFwMgD1kmVaSjaA+zgKIbfYnkOg1p0bPEcFLcDiZnkPEKM7CWIJDimupfbp4zXs/iZ0B0HOgj7UZPsLn0RWjfPly7tNaDjw1DTpFRXLs+AWQVEMGD8= ceph-654b0c8e-bac3-11ee-95b1-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-24T14:20:10.289 INFO:teuthology.orchestra.run.smithi123.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRBhvYBWmdATrSMvfWo6Pk/4oJflz89pkUJtpO7iZnUX31bWEnSrytuzacicjMV3MQ+QQkzGDZKaklWSetQGxNzyFpiRBVx+xlHz46XKSOd+8kYijwC/k0EGzuarszGkvIyvA/3EuKKs/QHQVcns1V3nPT+STZ6vjtJt0BlL5+x7VvYfnjyuwSgFLv7ipoMwCsV4IFbrxc1WrfKdNfjHf+lC3dzfI5gokxCrS+ZlCmzEP4vFrqqFKvIEvkDXsnZNuvZI4drAEIrhdUNYOmoiqOZZ1mH5WAv4K243R/lAFQbsSKkYXDAoEnYfDikIjFYalQUl7+VwbHIqNLi800cECQnsF6XO18CYeoMD85URhRmIxYnsHW4Yi24KVQq0ah/2knyYJMDJVkXwsJArFwMgD1kmVaSjaA+zgKIbfYnkOg1p0bPEcFLcDiZnkPEKM7CWIJDimupfbp4zXs/iZ0B0HOgj7UZPsLn0RWjfPly7tNaDjw1DTpFRXLs+AWQVEMGD8= ceph-654b0c8e-bac3-11ee-95b1-87774f69a715 2024-01-24T14:20:10.303 DEBUG:teuthology.orchestra.run.smithi194:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRBhvYBWmdATrSMvfWo6Pk/4oJflz89pkUJtpO7iZnUX31bWEnSrytuzacicjMV3MQ+QQkzGDZKaklWSetQGxNzyFpiRBVx+xlHz46XKSOd+8kYijwC/k0EGzuarszGkvIyvA/3EuKKs/QHQVcns1V3nPT+STZ6vjtJt0BlL5+x7VvYfnjyuwSgFLv7ipoMwCsV4IFbrxc1WrfKdNfjHf+lC3dzfI5gokxCrS+ZlCmzEP4vFrqqFKvIEvkDXsnZNuvZI4drAEIrhdUNYOmoiqOZZ1mH5WAv4K243R/lAFQbsSKkYXDAoEnYfDikIjFYalQUl7+VwbHIqNLi800cECQnsF6XO18CYeoMD85URhRmIxYnsHW4Yi24KVQq0ah/2knyYJMDJVkXwsJArFwMgD1kmVaSjaA+zgKIbfYnkOg1p0bPEcFLcDiZnkPEKM7CWIJDimupfbp4zXs/iZ0B0HOgj7UZPsLn0RWjfPly7tNaDjw1DTpFRXLs+AWQVEMGD8= ceph-654b0c8e-bac3-11ee-95b1-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-24T14:20:10.351 INFO:teuthology.orchestra.run.smithi194.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRBhvYBWmdATrSMvfWo6Pk/4oJflz89pkUJtpO7iZnUX31bWEnSrytuzacicjMV3MQ+QQkzGDZKaklWSetQGxNzyFpiRBVx+xlHz46XKSOd+8kYijwC/k0EGzuarszGkvIyvA/3EuKKs/QHQVcns1V3nPT+STZ6vjtJt0BlL5+x7VvYfnjyuwSgFLv7ipoMwCsV4IFbrxc1WrfKdNfjHf+lC3dzfI5gokxCrS+ZlCmzEP4vFrqqFKvIEvkDXsnZNuvZI4drAEIrhdUNYOmoiqOZZ1mH5WAv4K243R/lAFQbsSKkYXDAoEnYfDikIjFYalQUl7+VwbHIqNLi800cECQnsF6XO18CYeoMD85URhRmIxYnsHW4Yi24KVQq0ah/2knyYJMDJVkXwsJArFwMgD1kmVaSjaA+zgKIbfYnkOg1p0bPEcFLcDiZnkPEKM7CWIJDimupfbp4zXs/iZ0B0HOgj7UZPsLn0RWjfPly7tNaDjw1DTpFRXLs+AWQVEMGD8= ceph-654b0c8e-bac3-11ee-95b1-87774f69a715 2024-01-24T14:20:10.364 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-01-24T14:20:10.485 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:10 smithi019 ceph-mon[40663]: mgrmap e13: x(active, since 6s) 2024-01-24T14:20:10.485 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:10 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/355320621' entity='client.admin' 2024-01-24T14:20:10.610 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/mon.a/config 2024-01-24T14:20:11.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:11.915+0000 7f68f67c4700 1 -- 172.21.15.19:0/2321269049 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f68f0100850 msgr2=0x7f68f0100c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:11.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:11.915+0000 7f68f67c4700 1 --2- 172.21.15.19:0/2321269049 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f68f0100850 0x7f68f0100c70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f68e0009910 tx=0x7f68e0009c20 comp rx=0 tx=0).stop 2024-01-24T14:20:11.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:11.915+0000 7f68f67c4700 1 -- 172.21.15.19:0/2321269049 shutdown_connections 2024-01-24T14:20:11.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:11.915+0000 7f68f67c4700 1 --2- 172.21.15.19:0/2321269049 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f68f0100850 0x7f68f0100c70 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:11.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:11.915+0000 7f68f67c4700 1 -- 172.21.15.19:0/2321269049 >> 172.21.15.19:0/2321269049 conn(0x7f68f00fbdd0 msgr2=0x7f68f00fe230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:20:11.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:11.915+0000 7f68f67c4700 1 -- 172.21.15.19:0/2321269049 shutdown_connections 2024-01-24T14:20:11.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:11.915+0000 7f68f67c4700 1 -- 172.21.15.19:0/2321269049 wait complete. 2024-01-24T14:20:11.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:11.916+0000 7f68f67c4700 1 Processor -- start 2024-01-24T14:20:11.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:11.916+0000 7f68f67c4700 1 -- start start 2024-01-24T14:20:11.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:11.916+0000 7f68f67c4700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f68f0100850 0x7f68f0195960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:11.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:11.916+0000 7f68f67c4700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68f0195ea0 con 0x7f68f0100850 2024-01-24T14:20:11.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:11.917+0000 7f68effff700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f68f0100850 0x7f68f0195960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:11.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:11.917+0000 7f68effff700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f68f0100850 0x7f68f0195960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:42646/0 (socket says 172.21.15.19:42646) 2024-01-24T14:20:11.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:11.917+0000 7f68effff700 1 -- 172.21.15.19:0/2573610753 learned_addr learned my addr 172.21.15.19:0/2573610753 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:20:11.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:11.917+0000 7f68effff700 1 -- 172.21.15.19:0/2573610753 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e00095f0 con 0x7f68f0100850 2024-01-24T14:20:11.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:11.917+0000 7f68effff700 1 --2- 172.21.15.19:0/2573610753 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f68f0100850 0x7f68f0195960 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f68e0006010 tx=0x7f68e00057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:20:11.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:11.917+0000 7f68ed7fa700 1 -- 172.21.15.19:0/2573610753 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f68e001d070 con 0x7f68f0100850 2024-01-24T14:20:11.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:11.917+0000 7f68ed7fa700 1 -- 172.21.15.19:0/2573610753 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f68e00044e0 con 0x7f68f0100850 2024-01-24T14:20:11.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:11.917+0000 7f68ed7fa700 1 -- 172.21.15.19:0/2573610753 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f68e00039b0 con 0x7f68f0100850 2024-01-24T14:20:11.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:11.917+0000 7f68f67c4700 1 -- 172.21.15.19:0/2573610753 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68f01960a0 con 0x7f68f0100850 2024-01-24T14:20:11.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:11.917+0000 7f68f67c4700 1 -- 172.21.15.19:0/2573610753 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68f01964e0 con 0x7f68f0100850 2024-01-24T14:20:11.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:11.918+0000 7f68ed7fa700 1 -- 172.21.15.19:0/2573610753 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 13) v1 ==== 45324+0+0 (secure 0 0 0) 0x7f68e0003c70 con 0x7f68f0100850 2024-01-24T14:20:11.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:11.918+0000 7f68f67c4700 1 -- 172.21.15.19:0/2573610753 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68f018f360 con 0x7f68f0100850 2024-01-24T14:20:11.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:11.918+0000 7f68ed7fa700 1 --2- 172.21.15.19:0/2573610753 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f68d80383a0 0x7f68d803a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:11.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:11.918+0000 7f68ed7fa700 1 -- 172.21.15.19:0/2573610753 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f68e004c8d0 con 0x7f68f0100850 2024-01-24T14:20:11.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:11.918+0000 7f68ef7fe700 1 --2- 172.21.15.19:0/2573610753 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f68d80383a0 0x7f68d803a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:11.920 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:11.919+0000 7f68ef7fe700 1 --2- 172.21.15.19:0/2573610753 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f68d80383a0 0x7f68d803a860 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f68e4006fd0 tx=0x7f68e4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:20:11.923 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:11.923+0000 7f68ed7fa700 1 -- 172.21.15.19:0/2573610753 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f68e002a950 con 0x7f68f0100850 2024-01-24T14:20:12.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:12.139+0000 7f68f67c4700 1 -- 172.21.15.19:0/2573610753 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f68f002cc70 con 0x7f68f0100850 2024-01-24T14:20:12.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:12.143+0000 7f68ed7fa700 1 -- 172.21.15.19:0/2573610753 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f68e002a360 con 0x7f68f0100850 2024-01-24T14:20:12.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:12.145+0000 7f68f67c4700 1 -- 172.21.15.19:0/2573610753 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f68d80383a0 msgr2=0x7f68d803a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:12.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:12.145+0000 7f68f67c4700 1 --2- 172.21.15.19:0/2573610753 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f68d80383a0 0x7f68d803a860 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f68e4006fd0 tx=0x7f68e4006e40 comp rx=0 tx=0).stop 2024-01-24T14:20:12.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:12.145+0000 7f68f67c4700 1 -- 172.21.15.19:0/2573610753 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f68f0100850 msgr2=0x7f68f0195960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:12.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:12.146+0000 7f68f67c4700 1 --2- 172.21.15.19:0/2573610753 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f68f0100850 0x7f68f0195960 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f68e0006010 tx=0x7f68e00057c0 comp rx=0 tx=0).stop 2024-01-24T14:20:12.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:12.146+0000 7f68f67c4700 1 -- 172.21.15.19:0/2573610753 shutdown_connections 2024-01-24T14:20:12.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:12.146+0000 7f68f67c4700 1 --2- 172.21.15.19:0/2573610753 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f68d80383a0 0x7f68d803a860 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:12.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:12.146+0000 7f68f67c4700 1 --2- 172.21.15.19:0/2573610753 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f68f0100850 0x7f68f0195960 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:12.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:12.146+0000 7f68f67c4700 1 -- 172.21.15.19:0/2573610753 >> 172.21.15.19:0/2573610753 conn(0x7f68f00fbdd0 msgr2=0x7f68f00fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:20:12.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:12.146+0000 7f68f67c4700 1 -- 172.21.15.19:0/2573610753 shutdown_connections 2024-01-24T14:20:12.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:12.146+0000 7f68f67c4700 1 -- 172.21.15.19:0/2573610753 wait complete. 2024-01-24T14:20:12.669 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-01-24T14:20:12.669 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-01-24T14:20:12.930 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/mon.a/config 2024-01-24T14:20:13.311 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:13 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/2573610753' entity='client.admin' 2024-01-24T14:20:14.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:14.155+0000 7f974bcf2700 1 -- 172.21.15.19:0/1537781330 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f97440ff730 msgr2=0x7f97440ffb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:14.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:14.155+0000 7f974bcf2700 1 --2- 172.21.15.19:0/1537781330 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f97440ff730 0x7f97440ffb50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9740009910 tx=0x7f9740009c20 comp rx=0 tx=0).stop 2024-01-24T14:20:14.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:14.155+0000 7f974bcf2700 1 -- 172.21.15.19:0/1537781330 shutdown_connections 2024-01-24T14:20:14.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:14.155+0000 7f974bcf2700 1 --2- 172.21.15.19:0/1537781330 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f97440ff730 0x7f97440ffb50 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:14.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:14.155+0000 7f974bcf2700 1 -- 172.21.15.19:0/1537781330 >> 172.21.15.19:0/1537781330 conn(0x7f97440facd0 msgr2=0x7f97440fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:20:14.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:14.155+0000 7f974bcf2700 1 -- 172.21.15.19:0/1537781330 shutdown_connections 2024-01-24T14:20:14.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:14.155+0000 7f974bcf2700 1 -- 172.21.15.19:0/1537781330 wait complete. 2024-01-24T14:20:14.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:14.156+0000 7f974bcf2700 1 Processor -- start 2024-01-24T14:20:14.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:14.156+0000 7f974bcf2700 1 -- start start 2024-01-24T14:20:14.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:14.156+0000 7f974bcf2700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f97440ff730 0x7f9744195940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:14.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:14.156+0000 7f974bcf2700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9744195e80 con 0x7f97440ff730 2024-01-24T14:20:14.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:14.156+0000 7f9749a8e700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f97440ff730 0x7f9744195940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:14.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:14.156+0000 7f9749a8e700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f97440ff730 0x7f9744195940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:42656/0 (socket says 172.21.15.19:42656) 2024-01-24T14:20:14.158 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:14.156+0000 7f9749a8e700 1 -- 172.21.15.19:0/1007964449 learned_addr learned my addr 172.21.15.19:0/1007964449 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:20:14.158 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:14.157+0000 7f9749a8e700 1 -- 172.21.15.19:0/1007964449 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97400095f0 con 0x7f97440ff730 2024-01-24T14:20:14.158 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:14.157+0000 7f9749a8e700 1 --2- 172.21.15.19:0/1007964449 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f97440ff730 0x7f9744195940 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9740006010 tx=0x7f97400057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:20:14.159 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:14.157+0000 7f973affd700 1 -- 172.21.15.19:0/1007964449 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f974001d070 con 0x7f97440ff730 2024-01-24T14:20:14.159 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:14.157+0000 7f973affd700 1 -- 172.21.15.19:0/1007964449 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f97400044e0 con 0x7f97440ff730 2024-01-24T14:20:14.159 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:14.157+0000 7f974bcf2700 1 -- 172.21.15.19:0/1007964449 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9744196080 con 0x7f97440ff730 2024-01-24T14:20:14.159 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:14.157+0000 7f973affd700 1 -- 172.21.15.19:0/1007964449 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f97400039b0 con 0x7f97440ff730 2024-01-24T14:20:14.159 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:14.157+0000 7f974bcf2700 1 -- 172.21.15.19:0/1007964449 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97441964c0 con 0x7f97440ff730 2024-01-24T14:20:14.159 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:14.158+0000 7f973affd700 1 -- 172.21.15.19:0/1007964449 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 13) v1 ==== 45324+0+0 (secure 0 0 0) 0x7f974000baf0 con 0x7f97440ff730 2024-01-24T14:20:14.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:14.158+0000 7f973affd700 1 --2- 172.21.15.19:0/1007964449 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f97300383a0 0x7f973003a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:14.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:14.158+0000 7f973affd700 1 -- 172.21.15.19:0/1007964449 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f974004c2b0 con 0x7f97440ff730 2024-01-24T14:20:14.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:14.158+0000 7f974bcf2700 1 -- 172.21.15.19:0/1007964449 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f974418f250 con 0x7f97440ff730 2024-01-24T14:20:14.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:14.158+0000 7f974928d700 1 --2- 172.21.15.19:0/1007964449 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f97300383a0 0x7f973003a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:14.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:14.159+0000 7f974928d700 1 --2- 172.21.15.19:0/1007964449 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f97300383a0 0x7f973003a860 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9734006fd0 tx=0x7f9734006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:20:14.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:14.163+0000 7f973affd700 1 -- 172.21.15.19:0/1007964449 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f974002a540 con 0x7f97440ff730 2024-01-24T14:20:14.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:14.377+0000 7f974bcf2700 1 -- 172.21.15.19:0/1007964449 --> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f9744061210 con 0x7f97300383a0 2024-01-24T14:20:14.380 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:14.380+0000 7f973affd700 1 -- 172.21.15.19:0/1007964449 <== mgr.14152 v2:172.21.15.19:6800/1634350318 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9744061210 con 0x7f97300383a0 2024-01-24T14:20:14.382 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:14.382+0000 7f974bcf2700 1 -- 172.21.15.19:0/1007964449 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f97300383a0 msgr2=0x7f973003a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:14.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:14.382+0000 7f974bcf2700 1 --2- 172.21.15.19:0/1007964449 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f97300383a0 0x7f973003a860 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9734006fd0 tx=0x7f9734006e40 comp rx=0 tx=0).stop 2024-01-24T14:20:14.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:14.382+0000 7f974bcf2700 1 -- 172.21.15.19:0/1007964449 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f97440ff730 msgr2=0x7f9744195940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:14.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:14.382+0000 7f974bcf2700 1 --2- 172.21.15.19:0/1007964449 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f97440ff730 0x7f9744195940 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9740006010 tx=0x7f97400057c0 comp rx=0 tx=0).stop 2024-01-24T14:20:14.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:14.383+0000 7f974bcf2700 1 -- 172.21.15.19:0/1007964449 shutdown_connections 2024-01-24T14:20:14.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:14.383+0000 7f974bcf2700 1 --2- 172.21.15.19:0/1007964449 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f97300383a0 0x7f973003a860 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:14.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:14.383+0000 7f974bcf2700 1 --2- 172.21.15.19:0/1007964449 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f97440ff730 0x7f9744195940 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:14.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:14.383+0000 7f974bcf2700 1 -- 172.21.15.19:0/1007964449 >> 172.21.15.19:0/1007964449 conn(0x7f97440facd0 msgr2=0x7f97440687d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:20:14.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:14.383+0000 7f974bcf2700 1 -- 172.21.15.19:0/1007964449 shutdown_connections 2024-01-24T14:20:14.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:14.383+0000 7f974bcf2700 1 -- 172.21.15.19:0/1007964449 wait complete. 2024-01-24T14:20:14.676 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:14 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:14.676 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:14 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:14.676 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:14 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi019", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:20:14.676 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:14 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:14.676 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:14 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:20:14.676 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:14 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:14.677 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:14 smithi019 ceph-mon[40663]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:20:14.677 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:14 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:14.677 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:14 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:20:14.677 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:14 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:20:14.677 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:14 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:20:14.677 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:14 smithi019 ceph-mon[40663]: Updating smithi019:/etc/ceph/ceph.conf 2024-01-24T14:20:14.770 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi123 2024-01-24T14:20:14.770 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-01-24T14:20:14.770 DEBUG:teuthology.orchestra.run.smithi123:> dd of=/etc/ceph/ceph.conf 2024-01-24T14:20:14.791 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-01-24T14:20:14.791 DEBUG:teuthology.orchestra.run.smithi123:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-24T14:20:14.852 INFO:tasks.cephadm:Adding host smithi123 to orchestrator... 2024-01-24T14:20:14.852 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 -- ceph orch host add smithi123 2024-01-24T14:20:15.115 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/mon.a/config 2024-01-24T14:20:15.960 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:15 smithi019 ceph-mon[40663]: Updating smithi019:/var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/config/ceph.conf 2024-01-24T14:20:15.960 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:15 smithi019 ceph-mon[40663]: Updating smithi019:/etc/ceph/ceph.client.admin.keyring 2024-01-24T14:20:15.960 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:15 smithi019 ceph-mon[40663]: Updating smithi019:/var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/config/ceph.client.admin.keyring 2024-01-24T14:20:15.960 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:15 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:15.960 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:15 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:15.960 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:15 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:16.158 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:16.156+0000 7fd4c789d700 1 -- 172.21.15.19:0/2374200694 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd4c01067d0 msgr2=0x7fd4c0106bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:16.158 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:16.156+0000 7fd4c789d700 1 --2- 172.21.15.19:0/2374200694 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd4c01067d0 0x7fd4c0106bb0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc009910 tx=0x7fd4bc009c20 comp rx=0 tx=0).stop 2024-01-24T14:20:16.158 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:16.157+0000 7fd4c789d700 1 -- 172.21.15.19:0/2374200694 shutdown_connections 2024-01-24T14:20:16.159 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:16.157+0000 7fd4c789d700 1 --2- 172.21.15.19:0/2374200694 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd4c01067d0 0x7fd4c0106bb0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:16.159 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:16.157+0000 7fd4c789d700 1 -- 172.21.15.19:0/2374200694 >> 172.21.15.19:0/2374200694 conn(0x7fd4c0075240 msgr2=0x7fd4c0075650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:20:16.159 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:16.157+0000 7fd4c789d700 1 -- 172.21.15.19:0/2374200694 shutdown_connections 2024-01-24T14:20:16.159 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:16.157+0000 7fd4c789d700 1 -- 172.21.15.19:0/2374200694 wait complete. 2024-01-24T14:20:16.159 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:16.157+0000 7fd4c789d700 1 Processor -- start 2024-01-24T14:20:16.159 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:16.157+0000 7fd4c789d700 1 -- start start 2024-01-24T14:20:16.159 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:16.158+0000 7fd4c789d700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd4c01067d0 0x7fd4c00ff7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:16.159 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:16.158+0000 7fd4c789d700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c0101730 con 0x7fd4c01067d0 2024-01-24T14:20:16.159 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:16.158+0000 7fd4c5639700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd4c01067d0 0x7fd4c00ff7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:16.159 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:16.158+0000 7fd4c5639700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd4c01067d0 0x7fd4c00ff7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:42664/0 (socket says 172.21.15.19:42664) 2024-01-24T14:20:16.159 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:16.158+0000 7fd4c5639700 1 -- 172.21.15.19:0/2590151358 learned_addr learned my addr 172.21.15.19:0/2590151358 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:20:16.159 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:16.158+0000 7fd4c5639700 1 -- 172.21.15.19:0/2590151358 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4bc0095f0 con 0x7fd4c01067d0 2024-01-24T14:20:16.159 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:16.158+0000 7fd4c5639700 1 --2- 172.21.15.19:0/2590151358 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd4c01067d0 0x7fd4c00ff7b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc004f40 tx=0x7fd4bc005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:20:16.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:16.158+0000 7fd4b67fc700 1 -- 172.21.15.19:0/2590151358 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd4bc01d070 con 0x7fd4c01067d0 2024-01-24T14:20:16.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:16.158+0000 7fd4b67fc700 1 -- 172.21.15.19:0/2590151358 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd4bc004370 con 0x7fd4c01067d0 2024-01-24T14:20:16.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:16.158+0000 7fd4c789d700 1 -- 172.21.15.19:0/2590151358 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4c00ffcf0 con 0x7fd4c01067d0 2024-01-24T14:20:16.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:16.158+0000 7fd4b67fc700 1 -- 172.21.15.19:0/2590151358 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd4bc0039b0 con 0x7fd4c01067d0 2024-01-24T14:20:16.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:16.158+0000 7fd4c789d700 1 -- 172.21.15.19:0/2590151358 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4c0100170 con 0x7fd4c01067d0 2024-01-24T14:20:16.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:16.159+0000 7fd4b67fc700 1 -- 172.21.15.19:0/2590151358 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 13) v1 ==== 45324+0+0 (secure 0 0 0) 0x7fd4bc003c20 con 0x7fd4c01067d0 2024-01-24T14:20:16.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:16.159+0000 7fd4c789d700 1 -- 172.21.15.19:0/2590151358 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4c0105a80 con 0x7fd4c01067d0 2024-01-24T14:20:16.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:16.160+0000 7fd4b67fc700 1 --2- 172.21.15.19:0/2590151358 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fd4ac0383f0 0x7fd4ac03a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:16.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:16.160+0000 7fd4b67fc700 1 -- 172.21.15.19:0/2590151358 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd4bc04bff0 con 0x7fd4c01067d0 2024-01-24T14:20:16.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:16.160+0000 7fd4c4e38700 1 --2- 172.21.15.19:0/2590151358 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fd4ac0383f0 0x7fd4ac03a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:16.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:16.160+0000 7fd4c4e38700 1 --2- 172.21.15.19:0/2590151358 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fd4ac0383f0 0x7fd4ac03a8b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd4b0006fd0 tx=0x7fd4b0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:20:16.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:16.165+0000 7fd4b67fc700 1 -- 172.21.15.19:0/2590151358 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fd4bc07f120 con 0x7fd4c01067d0 2024-01-24T14:20:16.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:16.377+0000 7fd4c789d700 1 -- 172.21.15.19:0/2590151358 --> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi123", "target": ["mon-mgr", ""]}) v1 -- 0x7fd4c0100470 con 0x7fd4ac0383f0 2024-01-24T14:20:16.985 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:16 smithi019 ceph-mon[40663]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi123", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:20:17.985 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:17 smithi019 ceph-mon[40663]: Deploying cephadm binary to smithi123 2024-01-24T14:20:18.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:18.633+0000 7fd4b67fc700 1 -- 172.21.15.19:0/2590151358 <== mgr.14152 v2:172.21.15.19:6800/1634350318 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fd4c0100470 con 0x7fd4ac0383f0 2024-01-24T14:20:18.633 INFO:teuthology.orchestra.run.smithi019.stdout:Added host 'smithi123' with addr '172.21.15.123' 2024-01-24T14:20:18.635 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:18.635+0000 7fd4c789d700 1 -- 172.21.15.19:0/2590151358 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fd4ac0383f0 msgr2=0x7fd4ac03a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:18.636 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:18.635+0000 7fd4c789d700 1 --2- 172.21.15.19:0/2590151358 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fd4ac0383f0 0x7fd4ac03a8b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd4b0006fd0 tx=0x7fd4b0006e40 comp rx=0 tx=0).stop 2024-01-24T14:20:18.636 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:18.635+0000 7fd4c789d700 1 -- 172.21.15.19:0/2590151358 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd4c01067d0 msgr2=0x7fd4c00ff7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:18.636 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:18.635+0000 7fd4c789d700 1 --2- 172.21.15.19:0/2590151358 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd4c01067d0 0x7fd4c00ff7b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc004f40 tx=0x7fd4bc005020 comp rx=0 tx=0).stop 2024-01-24T14:20:18.636 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:18.636+0000 7fd4c789d700 1 -- 172.21.15.19:0/2590151358 shutdown_connections 2024-01-24T14:20:18.636 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:18.636+0000 7fd4c789d700 1 --2- 172.21.15.19:0/2590151358 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fd4ac0383f0 0x7fd4ac03a8b0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:18.636 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:18.636+0000 7fd4c789d700 1 --2- 172.21.15.19:0/2590151358 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd4c01067d0 0x7fd4c00ff7b0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:18.636 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:18.636+0000 7fd4c789d700 1 -- 172.21.15.19:0/2590151358 >> 172.21.15.19:0/2590151358 conn(0x7fd4c0075240 msgr2=0x7fd4c00fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:20:18.636 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:18.636+0000 7fd4c789d700 1 -- 172.21.15.19:0/2590151358 shutdown_connections 2024-01-24T14:20:18.636 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:18.636+0000 7fd4c789d700 1 -- 172.21.15.19:0/2590151358 wait complete. 2024-01-24T14:20:19.018 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 -- ceph orch host ls --format=json 2024-01-24T14:20:19.269 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/mon.a/config 2024-01-24T14:20:19.985 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:19 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:19.985 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:19 smithi019 ceph-mon[40663]: Added host smithi123 2024-01-24T14:20:19.985 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:19 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:20:19.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:19 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:19.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:19 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:20.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:20.366+0000 7f8f5566b700 1 -- 172.21.15.19:0/2020052713 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8f501067c0 msgr2=0x7f8f50106ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:20.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:20.366+0000 7f8f5566b700 1 --2- 172.21.15.19:0/2020052713 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8f501067c0 0x7f8f50106ba0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8f40009910 tx=0x7f8f40009c20 comp rx=0 tx=0).stop 2024-01-24T14:20:20.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:20.366+0000 7f8f5566b700 1 -- 172.21.15.19:0/2020052713 shutdown_connections 2024-01-24T14:20:20.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:20.366+0000 7f8f5566b700 1 --2- 172.21.15.19:0/2020052713 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8f501067c0 0x7f8f50106ba0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:20.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:20.366+0000 7f8f5566b700 1 -- 172.21.15.19:0/2020052713 >> 172.21.15.19:0/2020052713 conn(0x7f8f500fc2d0 msgr2=0x7f8f500fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:20:20.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:20.366+0000 7f8f5566b700 1 -- 172.21.15.19:0/2020052713 shutdown_connections 2024-01-24T14:20:20.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:20.366+0000 7f8f5566b700 1 -- 172.21.15.19:0/2020052713 wait complete. 2024-01-24T14:20:20.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:20.367+0000 7f8f5566b700 1 Processor -- start 2024-01-24T14:20:20.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:20.367+0000 7f8f5566b700 1 -- start start 2024-01-24T14:20:20.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:20.367+0000 7f8f5566b700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8f501067c0 0x7f8f5019ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:20.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:20.367+0000 7f8f5566b700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5019bf70 con 0x7f8f501067c0 2024-01-24T14:20:20.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:20.367+0000 7f8f4effd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8f501067c0 0x7f8f5019ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:20.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:20.367+0000 7f8f4effd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8f501067c0 0x7f8f5019ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:42678/0 (socket says 172.21.15.19:42678) 2024-01-24T14:20:20.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:20.367+0000 7f8f4effd700 1 -- 172.21.15.19:0/9223786 learned_addr learned my addr 172.21.15.19:0/9223786 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:20:20.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:20.368+0000 7f8f4effd700 1 -- 172.21.15.19:0/9223786 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f400095f0 con 0x7f8f501067c0 2024-01-24T14:20:20.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:20.368+0000 7f8f4effd700 1 --2- 172.21.15.19:0/9223786 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8f501067c0 0x7f8f5019ba30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8f40006010 tx=0x7f8f40004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:20:20.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:20.368+0000 7f8f4cff9700 1 -- 172.21.15.19:0/9223786 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8f4001d070 con 0x7f8f501067c0 2024-01-24T14:20:20.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:20.368+0000 7f8f4cff9700 1 -- 172.21.15.19:0/9223786 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8f4000bbb0 con 0x7f8f501067c0 2024-01-24T14:20:20.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:20.368+0000 7f8f5566b700 1 -- 172.21.15.19:0/9223786 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f5019c1d0 con 0x7f8f501067c0 2024-01-24T14:20:20.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:20.368+0000 7f8f4cff9700 1 -- 172.21.15.19:0/9223786 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8f40003930 con 0x7f8f501067c0 2024-01-24T14:20:20.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:20.368+0000 7f8f5566b700 1 -- 172.21.15.19:0/9223786 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f50195ab0 con 0x7f8f501067c0 2024-01-24T14:20:20.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:20.369+0000 7f8f4cff9700 1 -- 172.21.15.19:0/9223786 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 13) v1 ==== 45324+0+0 (secure 0 0 0) 0x7f8f40003b70 con 0x7f8f501067c0 2024-01-24T14:20:20.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:20.369+0000 7f8f5566b700 1 -- 172.21.15.19:0/9223786 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f50062370 con 0x7f8f501067c0 2024-01-24T14:20:20.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:20.369+0000 7f8f4cff9700 1 --2- 172.21.15.19:0/9223786 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f8f3c0383a0 0x7f8f3c03a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:20.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:20.369+0000 7f8f4cff9700 1 -- 172.21.15.19:0/9223786 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f8f4004c1f0 con 0x7f8f501067c0 2024-01-24T14:20:20.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:20.370+0000 7f8f467fc700 1 --2- 172.21.15.19:0/9223786 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f8f3c0383a0 0x7f8f3c03a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:20.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:20.370+0000 7f8f467fc700 1 --2- 172.21.15.19:0/9223786 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f8f3c0383a0 0x7f8f3c03a860 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8f38006fd0 tx=0x7f8f38006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:20:20.374 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:20.374+0000 7f8f4cff9700 1 -- 172.21.15.19:0/9223786 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f8f40029950 con 0x7f8f501067c0 2024-01-24T14:20:20.586 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:20.586+0000 7f8f5566b700 1 -- 172.21.15.19:0/9223786 --> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8f501964e0 con 0x7f8f3c0383a0 2024-01-24T14:20:20.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:20.587+0000 7f8f4cff9700 1 -- 172.21.15.19:0/9223786 <== mgr.14152 v2:172.21.15.19:6800/1634350318 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f8f501964e0 con 0x7f8f3c0383a0 2024-01-24T14:20:20.587 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:20:20.587 INFO:teuthology.orchestra.run.smithi019.stdout:[{"addr": "172.21.15.19", "hostname": "smithi019", "labels": [], "status": ""}, {"addr": "172.21.15.123", "hostname": "smithi123", "labels": [], "status": ""}] 2024-01-24T14:20:20.590 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:20.589+0000 7f8f5566b700 1 -- 172.21.15.19:0/9223786 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f8f3c0383a0 msgr2=0x7f8f3c03a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:20.590 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:20.589+0000 7f8f5566b700 1 --2- 172.21.15.19:0/9223786 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f8f3c0383a0 0x7f8f3c03a860 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8f38006fd0 tx=0x7f8f38006e40 comp rx=0 tx=0).stop 2024-01-24T14:20:20.590 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:20.589+0000 7f8f5566b700 1 -- 172.21.15.19:0/9223786 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8f501067c0 msgr2=0x7f8f5019ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:20.590 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:20.589+0000 7f8f5566b700 1 --2- 172.21.15.19:0/9223786 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8f501067c0 0x7f8f5019ba30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8f40006010 tx=0x7f8f40004ff0 comp rx=0 tx=0).stop 2024-01-24T14:20:20.590 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:20.590+0000 7f8f5566b700 1 -- 172.21.15.19:0/9223786 shutdown_connections 2024-01-24T14:20:20.590 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:20.590+0000 7f8f5566b700 1 --2- 172.21.15.19:0/9223786 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f8f3c0383a0 0x7f8f3c03a860 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:20.590 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:20.590+0000 7f8f5566b700 1 --2- 172.21.15.19:0/9223786 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8f501067c0 0x7f8f5019ba30 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:20.590 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:20.590+0000 7f8f5566b700 1 -- 172.21.15.19:0/9223786 >> 172.21.15.19:0/9223786 conn(0x7f8f500fc2d0 msgr2=0x7f8f50108c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:20:20.590 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:20.590+0000 7f8f5566b700 1 -- 172.21.15.19:0/9223786 shutdown_connections 2024-01-24T14:20:20.590 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:20.590+0000 7f8f5566b700 1 -- 172.21.15.19:0/9223786 wait complete. 2024-01-24T14:20:20.968 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi194 2024-01-24T14:20:20.969 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-01-24T14:20:20.969 DEBUG:teuthology.orchestra.run.smithi194:> dd of=/etc/ceph/ceph.conf 2024-01-24T14:20:20.990 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-01-24T14:20:20.990 DEBUG:teuthology.orchestra.run.smithi194:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-24T14:20:21.050 INFO:tasks.cephadm:Adding host smithi194 to orchestrator... 2024-01-24T14:20:21.050 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 -- ceph orch host add smithi194 2024-01-24T14:20:21.235 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:20 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:21.301 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/mon.a/config 2024-01-24T14:20:22.091 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:21 smithi019 ceph-mon[40663]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-24T14:20:22.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:22.301+0000 7fc0e59f0700 1 -- 172.21.15.19:0/3295068984 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc0e0105730 msgr2=0x7fc0e0105b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:22.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:22.301+0000 7fc0e59f0700 1 --2- 172.21.15.19:0/3295068984 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc0e0105730 0x7fc0e0105b10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc0d0009910 tx=0x7fc0d0009c20 comp rx=0 tx=0).stop 2024-01-24T14:20:22.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:22.301+0000 7fc0e59f0700 1 -- 172.21.15.19:0/3295068984 shutdown_connections 2024-01-24T14:20:22.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:22.301+0000 7fc0e59f0700 1 --2- 172.21.15.19:0/3295068984 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc0e0105730 0x7fc0e0105b10 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:22.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:22.301+0000 7fc0e59f0700 1 -- 172.21.15.19:0/3295068984 >> 172.21.15.19:0/3295068984 conn(0x7fc0e0075240 msgr2=0x7fc0e0075650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:20:22.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:22.301+0000 7fc0e59f0700 1 -- 172.21.15.19:0/3295068984 shutdown_connections 2024-01-24T14:20:22.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:22.301+0000 7fc0e59f0700 1 -- 172.21.15.19:0/3295068984 wait complete. 2024-01-24T14:20:22.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:22.302+0000 7fc0e59f0700 1 Processor -- start 2024-01-24T14:20:22.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:22.302+0000 7fc0e59f0700 1 -- start start 2024-01-24T14:20:22.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:22.302+0000 7fc0e59f0700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc0e0105730 0x7fc0e019bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:22.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:22.302+0000 7fc0e59f0700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e019c010 con 0x7fc0e0105730 2024-01-24T14:20:22.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:22.303+0000 7fc0deffd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc0e0105730 0x7fc0e019bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:22.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:22.303+0000 7fc0deffd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc0e0105730 0x7fc0e019bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:48390/0 (socket says 172.21.15.19:48390) 2024-01-24T14:20:22.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:22.303+0000 7fc0deffd700 1 -- 172.21.15.19:0/2144310191 learned_addr learned my addr 172.21.15.19:0/2144310191 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:20:22.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:22.303+0000 7fc0deffd700 1 -- 172.21.15.19:0/2144310191 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0d00095f0 con 0x7fc0e0105730 2024-01-24T14:20:22.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:22.303+0000 7fc0deffd700 1 --2- 172.21.15.19:0/2144310191 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc0e0105730 0x7fc0e019bad0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc0d0005850 tx=0x7fc0d0004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:20:22.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:22.303+0000 7fc0e49ee700 1 -- 172.21.15.19:0/2144310191 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc0d001d070 con 0x7fc0e0105730 2024-01-24T14:20:22.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:22.303+0000 7fc0e49ee700 1 -- 172.21.15.19:0/2144310191 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc0d0004370 con 0x7fc0e0105730 2024-01-24T14:20:22.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:22.303+0000 7fc0e59f0700 1 -- 172.21.15.19:0/2144310191 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0e019c210 con 0x7fc0e0105730 2024-01-24T14:20:22.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:22.303+0000 7fc0e49ee700 1 -- 172.21.15.19:0/2144310191 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc0d0003ab0 con 0x7fc0e0105730 2024-01-24T14:20:22.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:22.303+0000 7fc0e59f0700 1 -- 172.21.15.19:0/2144310191 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0e019c630 con 0x7fc0e0105730 2024-01-24T14:20:22.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:22.304+0000 7fc0e49ee700 1 -- 172.21.15.19:0/2144310191 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 13) v1 ==== 45324+0+0 (secure 0 0 0) 0x7fc0d000f460 con 0x7fc0e0105730 2024-01-24T14:20:22.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:22.304+0000 7fc0e59f0700 1 -- 172.21.15.19:0/2144310191 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0e0062370 con 0x7fc0e0105730 2024-01-24T14:20:22.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:22.304+0000 7fc0e49ee700 1 --2- 172.21.15.19:0/2144310191 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fc0c80383a0 0x7fc0c803a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:22.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:22.304+0000 7fc0e49ee700 1 -- 172.21.15.19:0/2144310191 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc0d004c190 con 0x7fc0e0105730 2024-01-24T14:20:22.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:22.305+0000 7fc0de7fc700 1 --2- 172.21.15.19:0/2144310191 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fc0c80383a0 0x7fc0c803a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:22.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:22.305+0000 7fc0de7fc700 1 --2- 172.21.15.19:0/2144310191 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fc0c80383a0 0x7fc0c803a860 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4006fd0 tx=0x7fc0d4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:20:22.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:22.309+0000 7fc0e49ee700 1 -- 172.21.15.19:0/2144310191 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fc0d000fd20 con 0x7fc0e0105730 2024-01-24T14:20:22.524 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:22.522+0000 7fc0e59f0700 1 -- 172.21.15.19:0/2144310191 --> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi194", "target": ["mon-mgr", ""]}) v1 -- 0x7fc0e0068650 con 0x7fc0c80383a0 2024-01-24T14:20:23.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:23 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:23.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:23 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:23.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:23 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:23.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:23 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:23.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:23 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi123", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:20:23.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:23 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:20:23.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:23 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:20:23.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:23 smithi019 ceph-mon[40663]: Updating smithi123:/etc/ceph/ceph.conf 2024-01-24T14:20:24.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:24 smithi019 ceph-mon[40663]: Updating smithi123:/var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/config/ceph.conf 2024-01-24T14:20:24.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:24 smithi019 ceph-mon[40663]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi194", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:20:24.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:24 smithi019 ceph-mon[40663]: Updating smithi123:/etc/ceph/ceph.client.admin.keyring 2024-01-24T14:20:24.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:24 smithi019 ceph-mon[40663]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:24.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:24 smithi019 ceph-mon[40663]: Updating smithi123:/var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/config/ceph.client.admin.keyring 2024-01-24T14:20:24.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:24 smithi019 ceph-mon[40663]: Deploying cephadm binary to smithi194 2024-01-24T14:20:24.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:24 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:24.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:24 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:24.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:24 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:24.882 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:24.881+0000 7fc0e49ee700 1 -- 172.21.15.19:0/2144310191 <== mgr.14152 v2:172.21.15.19:6800/1634350318 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fc0e0068650 con 0x7fc0c80383a0 2024-01-24T14:20:24.882 INFO:teuthology.orchestra.run.smithi019.stdout:Added host 'smithi194' with addr '172.21.15.194' 2024-01-24T14:20:24.885 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:24.884+0000 7fc0e59f0700 1 -- 172.21.15.19:0/2144310191 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fc0c80383a0 msgr2=0x7fc0c803a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:24.885 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:24.884+0000 7fc0e59f0700 1 --2- 172.21.15.19:0/2144310191 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fc0c80383a0 0x7fc0c803a860 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4006fd0 tx=0x7fc0d4006e40 comp rx=0 tx=0).stop 2024-01-24T14:20:24.885 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:24.884+0000 7fc0e59f0700 1 -- 172.21.15.19:0/2144310191 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc0e0105730 msgr2=0x7fc0e019bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:24.885 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:24.884+0000 7fc0e59f0700 1 --2- 172.21.15.19:0/2144310191 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc0e0105730 0x7fc0e019bad0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc0d0005850 tx=0x7fc0d0004ff0 comp rx=0 tx=0).stop 2024-01-24T14:20:24.885 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:24.884+0000 7fc0e59f0700 1 -- 172.21.15.19:0/2144310191 shutdown_connections 2024-01-24T14:20:24.885 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:24.884+0000 7fc0e59f0700 1 --2- 172.21.15.19:0/2144310191 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fc0c80383a0 0x7fc0c803a860 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:24.885 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:24.884+0000 7fc0e59f0700 1 --2- 172.21.15.19:0/2144310191 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc0e0105730 0x7fc0e019bad0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:24.886 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:24.884+0000 7fc0e59f0700 1 -- 172.21.15.19:0/2144310191 >> 172.21.15.19:0/2144310191 conn(0x7fc0e0075240 msgr2=0x7fc0e0067f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:20:24.886 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:24.884+0000 7fc0e59f0700 1 -- 172.21.15.19:0/2144310191 shutdown_connections 2024-01-24T14:20:24.886 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:24.884+0000 7fc0e59f0700 1 -- 172.21.15.19:0/2144310191 wait complete. 2024-01-24T14:20:25.294 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 -- ceph orch host ls --format=json 2024-01-24T14:20:25.547 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/mon.a/config 2024-01-24T14:20:26.235 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:25 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:26.235 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:25 smithi019 ceph-mon[40663]: Added host smithi194 2024-01-24T14:20:26.235 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:25 smithi019 ceph-mon[40663]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:26.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:25 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:20:26.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:25 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:26.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:25 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:26.887 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:26.886+0000 7fbff316b700 1 -- 172.21.15.19:0/4258306986 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbfec1067d0 msgr2=0x7fbfec106bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:26.887 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:26.886+0000 7fbff316b700 1 --2- 172.21.15.19:0/4258306986 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbfec1067d0 0x7fbfec106bb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8009910 tx=0x7fbfe8009c20 comp rx=0 tx=0).stop 2024-01-24T14:20:26.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:26.886+0000 7fbff316b700 1 -- 172.21.15.19:0/4258306986 shutdown_connections 2024-01-24T14:20:26.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:26.886+0000 7fbff316b700 1 --2- 172.21.15.19:0/4258306986 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbfec1067d0 0x7fbfec106bb0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:26.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:26.886+0000 7fbff316b700 1 -- 172.21.15.19:0/4258306986 >> 172.21.15.19:0/4258306986 conn(0x7fbfec075240 msgr2=0x7fbfec075650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:20:26.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:26.886+0000 7fbff316b700 1 -- 172.21.15.19:0/4258306986 shutdown_connections 2024-01-24T14:20:26.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:26.886+0000 7fbff316b700 1 -- 172.21.15.19:0/4258306986 wait complete. 2024-01-24T14:20:26.889 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:26.887+0000 7fbff316b700 1 Processor -- start 2024-01-24T14:20:26.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:26.887+0000 7fbff316b700 1 -- start start 2024-01-24T14:20:26.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:26.888+0000 7fbff316b700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbfec1067d0 0x7fbfec19ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:26.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:26.888+0000 7fbff316b700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfec19bf80 con 0x7fbfec1067d0 2024-01-24T14:20:26.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:26.888+0000 7fbff0f07700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbfec1067d0 0x7fbfec19ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:26.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:26.888+0000 7fbff0f07700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbfec1067d0 0x7fbfec19ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:48406/0 (socket says 172.21.15.19:48406) 2024-01-24T14:20:26.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:26.888+0000 7fbff0f07700 1 -- 172.21.15.19:0/2446992242 learned_addr learned my addr 172.21.15.19:0/2446992242 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:20:26.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:26.888+0000 7fbff0f07700 1 -- 172.21.15.19:0/2446992242 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe80095f0 con 0x7fbfec1067d0 2024-01-24T14:20:26.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:26.888+0000 7fbff0f07700 1 --2- 172.21.15.19:0/2446992242 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbfec1067d0 0x7fbfec19ba40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8006010 tx=0x7fbfe8004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:20:26.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:26.888+0000 7fbfe1ffb700 1 -- 172.21.15.19:0/2446992242 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbfe801d070 con 0x7fbfec1067d0 2024-01-24T14:20:26.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:26.888+0000 7fbff316b700 1 -- 172.21.15.19:0/2446992242 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfec19c1e0 con 0x7fbfec1067d0 2024-01-24T14:20:26.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:26.888+0000 7fbfe1ffb700 1 -- 172.21.15.19:0/2446992242 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbfe800bbb0 con 0x7fbfec1067d0 2024-01-24T14:20:26.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:26.888+0000 7fbfe1ffb700 1 -- 172.21.15.19:0/2446992242 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbfe8003930 con 0x7fbfec1067d0 2024-01-24T14:20:26.891 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:26.889+0000 7fbff316b700 1 -- 172.21.15.19:0/2446992242 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfec195ac0 con 0x7fbfec1067d0 2024-01-24T14:20:26.891 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:26.890+0000 7fbfe1ffb700 1 -- 172.21.15.19:0/2446992242 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 13) v1 ==== 45324+0+0 (secure 0 0 0) 0x7fbfe8003b70 con 0x7fbfec1067d0 2024-01-24T14:20:26.891 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:26.890+0000 7fbfe1ffb700 1 --2- 172.21.15.19:0/2446992242 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fbfd40383a0 0x7fbfd403a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:26.892 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:26.890+0000 7fbff316b700 1 -- 172.21.15.19:0/2446992242 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfec062370 con 0x7fbfec1067d0 2024-01-24T14:20:26.892 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:26.890+0000 7fbfe3fff700 1 --2- 172.21.15.19:0/2446992242 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fbfd40383a0 0x7fbfd403a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:26.892 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:26.890+0000 7fbfe1ffb700 1 -- 172.21.15.19:0/2446992242 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fbfe804bf90 con 0x7fbfec1067d0 2024-01-24T14:20:26.892 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:26.890+0000 7fbfe3fff700 1 --2- 172.21.15.19:0/2446992242 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fbfd40383a0 0x7fbfd403a860 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc006fd0 tx=0x7fbfdc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:20:26.896 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:26.895+0000 7fbfe1ffb700 1 -- 172.21.15.19:0/2446992242 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fbfe800b6f0 con 0x7fbfec1067d0 2024-01-24T14:20:27.111 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:27.110+0000 7fbff316b700 1 -- 172.21.15.19:0/2446992242 --> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbfec195cb0 con 0x7fbfd40383a0 2024-01-24T14:20:27.112 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:27.111+0000 7fbfe1ffb700 1 -- 172.21.15.19:0/2446992242 <== mgr.14152 v2:172.21.15.19:6800/1634350318 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+240 (secure 0 0 0) 0x7fbfec195cb0 con 0x7fbfd40383a0 2024-01-24T14:20:27.112 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:20:27.112 INFO:teuthology.orchestra.run.smithi019.stdout:[{"addr": "172.21.15.19", "hostname": "smithi019", "labels": [], "status": ""}, {"addr": "172.21.15.123", "hostname": "smithi123", "labels": [], "status": ""}, {"addr": "172.21.15.194", "hostname": "smithi194", "labels": [], "status": ""}] 2024-01-24T14:20:27.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:27.113+0000 7fbff316b700 1 -- 172.21.15.19:0/2446992242 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fbfd40383a0 msgr2=0x7fbfd403a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:27.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:27.113+0000 7fbff316b700 1 --2- 172.21.15.19:0/2446992242 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fbfd40383a0 0x7fbfd403a860 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc006fd0 tx=0x7fbfdc006e40 comp rx=0 tx=0).stop 2024-01-24T14:20:27.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:27.113+0000 7fbff316b700 1 -- 172.21.15.19:0/2446992242 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbfec1067d0 msgr2=0x7fbfec19ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:27.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:27.113+0000 7fbff316b700 1 --2- 172.21.15.19:0/2446992242 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbfec1067d0 0x7fbfec19ba40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8006010 tx=0x7fbfe8004ff0 comp rx=0 tx=0).stop 2024-01-24T14:20:27.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:27.113+0000 7fbff316b700 1 -- 172.21.15.19:0/2446992242 shutdown_connections 2024-01-24T14:20:27.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:27.113+0000 7fbff316b700 1 --2- 172.21.15.19:0/2446992242 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fbfd40383a0 0x7fbfd403a860 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:27.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:27.113+0000 7fbff316b700 1 --2- 172.21.15.19:0/2446992242 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbfec1067d0 0x7fbfec19ba40 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:27.115 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:27.113+0000 7fbff316b700 1 -- 172.21.15.19:0/2446992242 >> 172.21.15.19:0/2446992242 conn(0x7fbfec075240 msgr2=0x7fbfec0fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:20:27.115 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:27.113+0000 7fbff316b700 1 -- 172.21.15.19:0/2446992242 shutdown_connections 2024-01-24T14:20:27.115 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:27.113+0000 7fbff316b700 1 -- 172.21.15.19:0/2446992242 wait complete. 2024-01-24T14:20:27.400 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:27 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:27.492 INFO:tasks.cephadm:Setting crush tunables to default 2024-01-24T14:20:27.492 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 -- ceph osd crush tunables default 2024-01-24T14:20:27.741 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/mon.a/config 2024-01-24T14:20:28.485 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:28 smithi019 ceph-mon[40663]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:28.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:28 smithi019 ceph-mon[40663]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-24T14:20:28.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:28.769+0000 7fedb1dde700 1 -- 172.21.15.19:0/901840883 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fedac1067d0 msgr2=0x7fedac106bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:28.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:28.769+0000 7fedb1dde700 1 --2- 172.21.15.19:0/901840883 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fedac1067d0 0x7fedac106bb0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fed9c009910 tx=0x7fed9c009c20 comp rx=0 tx=0).stop 2024-01-24T14:20:28.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:28.769+0000 7fedb1dde700 1 -- 172.21.15.19:0/901840883 shutdown_connections 2024-01-24T14:20:28.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:28.769+0000 7fedb1dde700 1 --2- 172.21.15.19:0/901840883 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fedac1067d0 0x7fedac106bb0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:28.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:28.769+0000 7fedb1dde700 1 -- 172.21.15.19:0/901840883 >> 172.21.15.19:0/901840883 conn(0x7fedac075240 msgr2=0x7fedac075650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:20:28.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:28.769+0000 7fedb1dde700 1 -- 172.21.15.19:0/901840883 shutdown_connections 2024-01-24T14:20:28.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:28.769+0000 7fedb1dde700 1 -- 172.21.15.19:0/901840883 wait complete. 2024-01-24T14:20:28.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:28.770+0000 7fedb1dde700 1 Processor -- start 2024-01-24T14:20:28.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:28.770+0000 7fedb1dde700 1 -- start start 2024-01-24T14:20:28.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:28.770+0000 7fedb1dde700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fedac1067d0 0x7fedac0ff7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:28.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:28.770+0000 7fedb1dde700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedac101730 con 0x7fedac1067d0 2024-01-24T14:20:28.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:28.770+0000 7fedab7fe700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fedac1067d0 0x7fedac0ff7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:28.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:28.770+0000 7fedab7fe700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fedac1067d0 0x7fedac0ff7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:48422/0 (socket says 172.21.15.19:48422) 2024-01-24T14:20:28.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:28.770+0000 7fedab7fe700 1 -- 172.21.15.19:0/932515429 learned_addr learned my addr 172.21.15.19:0/932515429 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:20:28.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:28.771+0000 7fedab7fe700 1 -- 172.21.15.19:0/932515429 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed9c0095f0 con 0x7fedac1067d0 2024-01-24T14:20:28.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:28.771+0000 7fedab7fe700 1 --2- 172.21.15.19:0/932515429 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fedac1067d0 0x7fedac0ff7b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fed9c004f40 tx=0x7fed9c005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:20:28.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:28.771+0000 7feda8ff9700 1 -- 172.21.15.19:0/932515429 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fed9c01d070 con 0x7fedac1067d0 2024-01-24T14:20:28.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:28.771+0000 7fedb1dde700 1 -- 172.21.15.19:0/932515429 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedac0ffcf0 con 0x7fedac1067d0 2024-01-24T14:20:28.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:28.771+0000 7feda8ff9700 1 -- 172.21.15.19:0/932515429 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fed9c004370 con 0x7fedac1067d0 2024-01-24T14:20:28.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:28.771+0000 7feda8ff9700 1 -- 172.21.15.19:0/932515429 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fed9c0039b0 con 0x7fedac1067d0 2024-01-24T14:20:28.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:28.771+0000 7fedb1dde700 1 -- 172.21.15.19:0/932515429 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedac100170 con 0x7fedac1067d0 2024-01-24T14:20:28.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:28.772+0000 7feda8ff9700 1 -- 172.21.15.19:0/932515429 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 13) v1 ==== 45324+0+0 (secure 0 0 0) 0x7fed9c003c70 con 0x7fedac1067d0 2024-01-24T14:20:28.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:28.772+0000 7fedb1dde700 1 -- 172.21.15.19:0/932515429 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedac105a80 con 0x7fedac1067d0 2024-01-24T14:20:28.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:28.772+0000 7feda8ff9700 1 --2- 172.21.15.19:0/932515429 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fed980383a0 0x7fed9803a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:28.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:28.772+0000 7feda8ff9700 1 -- 172.21.15.19:0/932515429 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fed9c04c060 con 0x7fedac1067d0 2024-01-24T14:20:28.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:28.773+0000 7fedaaffd700 1 --2- 172.21.15.19:0/932515429 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fed980383a0 0x7fed9803a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:28.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:28.773+0000 7fedaaffd700 1 --2- 172.21.15.19:0/932515429 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fed980383a0 0x7fed9803a860 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fed94006fd0 tx=0x7fed94006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:20:28.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:28.777+0000 7feda8ff9700 1 -- 172.21.15.19:0/932515429 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fed9c02aba0 con 0x7fedac1067d0 2024-01-24T14:20:29.029 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:29.028+0000 7fedb1dde700 1 -- 172.21.15.19:0/932515429 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fedac062370 con 0x7fedac1067d0 2024-01-24T14:20:29.486 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:29.485+0000 7feda8ff9700 1 -- 172.21.15.19:0/932515429 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7fed9c026070 con 0x7fedac1067d0 2024-01-24T14:20:29.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:29.489+0000 7fedb1dde700 1 -- 172.21.15.19:0/932515429 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fed980383a0 msgr2=0x7fed9803a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:29.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:29.489+0000 7fedb1dde700 1 --2- 172.21.15.19:0/932515429 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fed980383a0 0x7fed9803a860 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fed94006fd0 tx=0x7fed94006e40 comp rx=0 tx=0).stop 2024-01-24T14:20:29.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:29.489+0000 7fedb1dde700 1 -- 172.21.15.19:0/932515429 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fedac1067d0 msgr2=0x7fedac0ff7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:29.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:29.489+0000 7fedb1dde700 1 --2- 172.21.15.19:0/932515429 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fedac1067d0 0x7fedac0ff7b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fed9c004f40 tx=0x7fed9c005020 comp rx=0 tx=0).stop 2024-01-24T14:20:29.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:29.489+0000 7fedb1dde700 1 -- 172.21.15.19:0/932515429 shutdown_connections 2024-01-24T14:20:29.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:29.489+0000 7fedb1dde700 1 --2- 172.21.15.19:0/932515429 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fed980383a0 0x7fed9803a860 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:29.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:29.489+0000 7fedb1dde700 1 --2- 172.21.15.19:0/932515429 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fedac1067d0 0x7fedac0ff7b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:29.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:29.489+0000 7fedb1dde700 1 -- 172.21.15.19:0/932515429 >> 172.21.15.19:0/932515429 conn(0x7fedac075240 msgr2=0x7fedac0fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:20:29.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:29.489+0000 7fedb1dde700 1 -- 172.21.15.19:0/932515429 shutdown_connections 2024-01-24T14:20:29.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:29.489+0000 7fedb1dde700 1 -- 172.21.15.19:0/932515429 wait complete. 2024-01-24T14:20:29.491 INFO:teuthology.orchestra.run.smithi019.stderr:adjusted tunables profile to default 2024-01-24T14:20:29.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:29 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:29.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:29 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:29.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:29 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:29.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:29 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:29.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:29 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi194", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:20:29.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:29 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:20:29.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:29 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:20:29.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:29 smithi019 ceph-mon[40663]: Updating smithi194:/etc/ceph/ceph.conf 2024-01-24T14:20:29.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:29 smithi019 ceph-mon[40663]: Updating smithi194:/var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/config/ceph.conf 2024-01-24T14:20:29.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:29 smithi019 ceph-mon[40663]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:29.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:29 smithi019 ceph-mon[40663]: Updating smithi194:/etc/ceph/ceph.client.admin.keyring 2024-01-24T14:20:29.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:29 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/932515429' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-01-24T14:20:29.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:29 smithi019 ceph-mon[40663]: Updating smithi194:/var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/config/ceph.client.admin.keyring 2024-01-24T14:20:29.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:29 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:29.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:29 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:29.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:29 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:29.867 INFO:tasks.cephadm:Adding mon.a on smithi019 2024-01-24T14:20:29.867 INFO:tasks.cephadm:Adding mon.b on smithi123 2024-01-24T14:20:29.867 INFO:tasks.cephadm:Adding mon.c on smithi194 2024-01-24T14:20:29.868 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 -- ceph orch apply mon '3;smithi019:172.21.15.19=a;smithi123:172.21.15.123=b;smithi194:172.21.15.194=c' 2024-01-24T14:20:30.109 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/config/ceph.conf 2024-01-24T14:20:30.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:30 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/932515429' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-01-24T14:20:30.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:30 smithi019 ceph-mon[40663]: osdmap e4: 0 total, 0 up, 0 in 2024-01-24T14:20:31.067 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:31.065+0000 7f67e97b7700 1 -- 172.21.15.194:0/3862866023 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67e4105730 msgr2=0x7f67e4105b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:31.067 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:31.065+0000 7f67e97b7700 1 --2- 172.21.15.194:0/3862866023 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67e4105730 0x7f67e4105b10 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f67d8009910 tx=0x7f67d8009c20 comp rx=0 tx=0).stop 2024-01-24T14:20:31.067 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:31.065+0000 7f67e97b7700 1 -- 172.21.15.194:0/3862866023 shutdown_connections 2024-01-24T14:20:31.067 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:31.065+0000 7f67e97b7700 1 --2- 172.21.15.194:0/3862866023 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67e4105730 0x7f67e4105b10 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:31.068 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:31.065+0000 7f67e97b7700 1 -- 172.21.15.194:0/3862866023 >> 172.21.15.194:0/3862866023 conn(0x7f67e4075240 msgr2=0x7f67e4075650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:20:31.068 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:31.066+0000 7f67e97b7700 1 -- 172.21.15.194:0/3862866023 shutdown_connections 2024-01-24T14:20:31.068 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:31.066+0000 7f67e97b7700 1 -- 172.21.15.194:0/3862866023 wait complete. 2024-01-24T14:20:31.068 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:31.066+0000 7f67e97b7700 1 Processor -- start 2024-01-24T14:20:31.068 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:31.066+0000 7f67e97b7700 1 -- start start 2024-01-24T14:20:31.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:31.066+0000 7f67e97b7700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67e4105730 0x7f67e419bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:31.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:31.066+0000 7f67e97b7700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67e419c010 con 0x7f67e4105730 2024-01-24T14:20:31.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:31.067+0000 7f67e2ffd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67e4105730 0x7f67e419bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:31.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:31.067+0000 7f67e2ffd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67e4105730 0x7f67e419bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.194:34566/0 (socket says 172.21.15.194:34566) 2024-01-24T14:20:31.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:31.067+0000 7f67e2ffd700 1 -- 172.21.15.194:0/3995004756 learned_addr learned my addr 172.21.15.194:0/3995004756 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-24T14:20:31.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:31.067+0000 7f67e2ffd700 1 -- 172.21.15.194:0/3995004756 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67d80095f0 con 0x7f67e4105730 2024-01-24T14:20:31.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:31.067+0000 7f67e2ffd700 1 --2- 172.21.15.194:0/3995004756 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67e4105730 0x7f67e419bad0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f67d8005850 tx=0x7f67d8004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:20:31.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:31.068+0000 7f67cbfff700 1 -- 172.21.15.194:0/3995004756 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f67d801d070 con 0x7f67e4105730 2024-01-24T14:20:31.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:31.068+0000 7f67e97b7700 1 -- 172.21.15.194:0/3995004756 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67e419c210 con 0x7f67e4105730 2024-01-24T14:20:31.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:31.068+0000 7f67cbfff700 1 -- 172.21.15.194:0/3995004756 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f67d8004370 con 0x7f67e4105730 2024-01-24T14:20:31.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:31.068+0000 7f67cbfff700 1 -- 172.21.15.194:0/3995004756 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f67d8003ab0 con 0x7f67e4105730 2024-01-24T14:20:31.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:31.068+0000 7f67e97b7700 1 -- 172.21.15.194:0/3995004756 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67e419c630 con 0x7f67e4105730 2024-01-24T14:20:31.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:31.069+0000 7f67e97b7700 1 -- 172.21.15.194:0/3995004756 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67e4062370 con 0x7f67e4105730 2024-01-24T14:20:31.072 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:31.069+0000 7f67cbfff700 1 -- 172.21.15.194:0/3995004756 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 13) v1 ==== 45324+0+0 (secure 0 0 0) 0x7f67d800b6f0 con 0x7f67e4105730 2024-01-24T14:20:31.072 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:31.069+0000 7f67cbfff700 1 --2- 172.21.15.194:0/3995004756 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f67cc0383a0 0x7f67cc03a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:31.072 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:31.069+0000 7f67cbfff700 1 -- 172.21.15.194:0/3995004756 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f67d804c100 con 0x7f67e4105730 2024-01-24T14:20:31.072 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:31.069+0000 7f67e27fc700 1 --2- 172.21.15.194:0/3995004756 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f67cc0383a0 0x7f67cc03a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:31.072 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:31.070+0000 7f67e27fc700 1 --2- 172.21.15.194:0/3995004756 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f67cc0383a0 0x7f67cc03a860 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f67d4006fd0 tx=0x7f67d4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:20:31.075 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:31.074+0000 7f67cbfff700 1 -- 172.21.15.194:0/3995004756 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f67d802a590 con 0x7f67e4105730 2024-01-24T14:20:31.274 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:31.273+0000 7f67e97b7700 1 -- 172.21.15.194:0/3995004756 --> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi019:172.21.15.19=a;smithi123:172.21.15.123=b;smithi194:172.21.15.194=c", "target": ["mon-mgr", ""]}) v1 -- 0x7f67e4068710 con 0x7f67cc0383a0 2024-01-24T14:20:31.281 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:31.281+0000 7f67cbfff700 1 -- 172.21.15.194:0/3995004756 <== mgr.14152 v2:172.21.15.19:6800/1634350318 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f67e4068710 con 0x7f67cc0383a0 2024-01-24T14:20:31.282 INFO:teuthology.orchestra.run.smithi194.stdout:Scheduled mon update... 2024-01-24T14:20:31.283 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:31.282+0000 7f67e97b7700 1 -- 172.21.15.194:0/3995004756 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f67cc0383a0 msgr2=0x7f67cc03a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:31.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:31.282+0000 7f67e97b7700 1 --2- 172.21.15.194:0/3995004756 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f67cc0383a0 0x7f67cc03a860 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f67d4006fd0 tx=0x7f67d4006e40 comp rx=0 tx=0).stop 2024-01-24T14:20:31.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:31.282+0000 7f67e97b7700 1 -- 172.21.15.194:0/3995004756 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67e4105730 msgr2=0x7f67e419bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:31.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:31.282+0000 7f67e97b7700 1 --2- 172.21.15.194:0/3995004756 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67e4105730 0x7f67e419bad0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f67d8005850 tx=0x7f67d8004ff0 comp rx=0 tx=0).stop 2024-01-24T14:20:31.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:31.282+0000 7f67e97b7700 1 -- 172.21.15.194:0/3995004756 shutdown_connections 2024-01-24T14:20:31.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:31.282+0000 7f67e97b7700 1 --2- 172.21.15.194:0/3995004756 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f67cc0383a0 0x7f67cc03a860 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:31.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:31.282+0000 7f67e97b7700 1 --2- 172.21.15.194:0/3995004756 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67e4105730 0x7f67e419bad0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:31.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:31.282+0000 7f67e97b7700 1 -- 172.21.15.194:0/3995004756 >> 172.21.15.194:0/3995004756 conn(0x7f67e4075240 msgr2=0x7f67e4067f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:20:31.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:31.282+0000 7f67e97b7700 1 -- 172.21.15.194:0/3995004756 shutdown_connections 2024-01-24T14:20:31.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:31.282+0000 7f67e97b7700 1 -- 172.21.15.194:0/3995004756 wait complete. 2024-01-24T14:20:31.675 DEBUG:teuthology.orchestra.run.smithi123:mon.b> sudo journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mon.b.service 2024-01-24T14:20:31.678 DEBUG:teuthology.orchestra.run.smithi194:mon.c> sudo journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mon.c.service 2024-01-24T14:20:31.680 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-01-24T14:20:31.681 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 -- ceph mon dump -f json 2024-01-24T14:20:31.711 INFO:journalctl@ceph.mon.c.smithi194.stdout:-- Logs begin at Wed 2024-01-24 14:04:01 UTC. -- 2024-01-24T14:20:31.715 INFO:journalctl@ceph.mon.b.smithi123.stdout:-- Logs begin at Wed 2024-01-24 14:03:45 UTC. -- 2024-01-24T14:20:32.027 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/config/ceph.conf 2024-01-24T14:20:32.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:32 smithi019 ceph-mon[40663]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:32.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:32 smithi019 ceph-mon[40663]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi019:172.21.15.19=a;smithi123:172.21.15.123=b;smithi194:172.21.15.194=c", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:20:32.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:32 smithi019 ceph-mon[40663]: Saving service mon spec with placement smithi019:172.21.15.19=a;smithi123:172.21.15.123=b;smithi194:172.21.15.194=c;count:3 2024-01-24T14:20:32.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:32 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:32.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:32 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:20:32.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:32 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:20:32.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:32 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:20:32.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:32 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:32.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:32 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-24T14:20:32.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:32 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:20:32.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:32 smithi019 ceph-mon[40663]: Deploying daemon mon.c on smithi194 2024-01-24T14:20:33.701 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:33.700+0000 7fcff3904700 1 -- 172.21.15.194:0/2168251522 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcfec1067d0 msgr2=0x7fcfec106bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:33.701 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:33.700+0000 7fcff3904700 1 --2- 172.21.15.194:0/2168251522 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcfec1067d0 0x7fcfec106bb0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8009910 tx=0x7fcfe8009c20 comp rx=0 tx=0).stop 2024-01-24T14:20:33.702 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:33.700+0000 7fcff3904700 1 -- 172.21.15.194:0/2168251522 shutdown_connections 2024-01-24T14:20:33.702 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:33.700+0000 7fcff3904700 1 --2- 172.21.15.194:0/2168251522 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcfec1067d0 0x7fcfec106bb0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:33.702 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:33.700+0000 7fcff3904700 1 -- 172.21.15.194:0/2168251522 >> 172.21.15.194:0/2168251522 conn(0x7fcfec075240 msgr2=0x7fcfec075650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:20:33.702 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:33.700+0000 7fcff3904700 1 -- 172.21.15.194:0/2168251522 shutdown_connections 2024-01-24T14:20:33.702 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:33.700+0000 7fcff3904700 1 -- 172.21.15.194:0/2168251522 wait complete. 2024-01-24T14:20:33.702 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:33.700+0000 7fcff3904700 1 Processor -- start 2024-01-24T14:20:33.702 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:33.700+0000 7fcff3904700 1 -- start start 2024-01-24T14:20:33.703 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:33.700+0000 7fcff3904700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcfec1067d0 0x7fcfec0ff7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:33.703 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:33.701+0000 7fcff3904700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfec101730 con 0x7fcfec1067d0 2024-01-24T14:20:33.703 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:33.701+0000 7fcff16a0700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcfec1067d0 0x7fcfec0ff7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:33.703 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:33.701+0000 7fcff16a0700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcfec1067d0 0x7fcfec0ff7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.194:34594/0 (socket says 172.21.15.194:34594) 2024-01-24T14:20:33.704 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:33.701+0000 7fcff16a0700 1 -- 172.21.15.194:0/1173817460 learned_addr learned my addr 172.21.15.194:0/1173817460 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-24T14:20:33.704 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:33.701+0000 7fcff16a0700 1 -- 172.21.15.194:0/1173817460 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfe80095f0 con 0x7fcfec1067d0 2024-01-24T14:20:33.704 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:33.701+0000 7fcff16a0700 1 --2- 172.21.15.194:0/1173817460 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcfec1067d0 0x7fcfec0ff7b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8004f40 tx=0x7fcfe8005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:20:33.704 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:33.702+0000 7fcfe27fc700 1 -- 172.21.15.194:0/1173817460 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcfe801d070 con 0x7fcfec1067d0 2024-01-24T14:20:33.704 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:33.702+0000 7fcff3904700 1 -- 172.21.15.194:0/1173817460 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfec0ffcf0 con 0x7fcfec1067d0 2024-01-24T14:20:33.704 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:33.702+0000 7fcfe27fc700 1 -- 172.21.15.194:0/1173817460 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcfe8004370 con 0x7fcfec1067d0 2024-01-24T14:20:33.704 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:33.702+0000 7fcfe27fc700 1 -- 172.21.15.194:0/1173817460 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcfe80039b0 con 0x7fcfec1067d0 2024-01-24T14:20:33.704 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:33.702+0000 7fcff3904700 1 -- 172.21.15.194:0/1173817460 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfec100170 con 0x7fcfec1067d0 2024-01-24T14:20:33.705 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:33.703+0000 7fcfe27fc700 1 -- 172.21.15.194:0/1173817460 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 13) v1 ==== 45324+0+0 (secure 0 0 0) 0x7fcfe8003c70 con 0x7fcfec1067d0 2024-01-24T14:20:33.705 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:33.703+0000 7fcff3904700 1 -- 172.21.15.194:0/1173817460 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfec105a80 con 0x7fcfec1067d0 2024-01-24T14:20:33.705 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:33.703+0000 7fcfe27fc700 1 --2- 172.21.15.194:0/1173817460 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fcfd80383a0 0x7fcfd803a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:33.705 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:33.703+0000 7fcfe27fc700 1 -- 172.21.15.194:0/1173817460 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fcfe804c150 con 0x7fcfec1067d0 2024-01-24T14:20:33.705 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:33.703+0000 7fcff0e9f700 1 --2- 172.21.15.194:0/1173817460 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fcfd80383a0 0x7fcfd803a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:33.705 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:33.704+0000 7fcff0e9f700 1 --2- 172.21.15.194:0/1173817460 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fcfd80383a0 0x7fcfd803a860 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc006fd0 tx=0x7fcfdc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:20:33.709 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:33.708+0000 7fcfe27fc700 1 -- 172.21.15.194:0/1173817460 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fcfe8029360 con 0x7fcfec1067d0 2024-01-24T14:20:34.008 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:34.006+0000 7fcff3904700 1 -- 172.21.15.194:0/1173817460 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcfec062370 con 0x7fcfec1067d0 2024-01-24T14:20:34.008 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:34.006+0000 7fcfe27fc700 1 -- 172.21.15.194:0/1173817460 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+736 (secure 0 0 0) 0x7fcfe8026030 con 0x7fcfec1067d0 2024-01-24T14:20:34.008 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-24T14:20:34.009 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"654b0c8e-bac3-11ee-95b1-87774f69a715","modified":"2024-01-24T14:19:08.559591Z","created":"2024-01-24T14:19:08.559591Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-24T14:20:34.010 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:34.008+0000 7fcff3904700 1 -- 172.21.15.194:0/1173817460 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fcfd80383a0 msgr2=0x7fcfd803a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:34.010 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:34.008+0000 7fcff3904700 1 --2- 172.21.15.194:0/1173817460 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fcfd80383a0 0x7fcfd803a860 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc006fd0 tx=0x7fcfdc006e40 comp rx=0 tx=0).stop 2024-01-24T14:20:34.010 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:34.009+0000 7fcff3904700 1 -- 172.21.15.194:0/1173817460 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcfec1067d0 msgr2=0x7fcfec0ff7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:34.010 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:34.009+0000 7fcff3904700 1 --2- 172.21.15.194:0/1173817460 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcfec1067d0 0x7fcfec0ff7b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8004f40 tx=0x7fcfe8005020 comp rx=0 tx=0).stop 2024-01-24T14:20:34.010 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:34.009+0000 7fcff3904700 1 -- 172.21.15.194:0/1173817460 shutdown_connections 2024-01-24T14:20:34.010 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:34.009+0000 7fcff3904700 1 --2- 172.21.15.194:0/1173817460 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fcfd80383a0 0x7fcfd803a860 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:34.010 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:34.009+0000 7fcff3904700 1 --2- 172.21.15.194:0/1173817460 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fcfec1067d0 0x7fcfec0ff7b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:34.010 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:34.009+0000 7fcff3904700 1 -- 172.21.15.194:0/1173817460 >> 172.21.15.194:0/1173817460 conn(0x7fcfec075240 msgr2=0x7fcfec0fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:20:34.010 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:34.009+0000 7fcff3904700 1 -- 172.21.15.194:0/1173817460 shutdown_connections 2024-01-24T14:20:34.011 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:34.009+0000 7fcff3904700 1 -- 172.21.15.194:0/1173817460 wait complete. 2024-01-24T14:20:34.011 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-24T14:20:34.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:34 smithi019 ceph-mon[40663]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:34.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:34 smithi019 ceph-mon[40663]: from='client.? 172.21.15.194:0/1173817460' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-24T14:20:34.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 systemd[1]: Started Ceph mon.c for 654b0c8e-bac3-11ee-95b1-87774f69a715. 2024-01-24T14:20:34.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-24T14:20:34.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-24T14:20:34.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: rocksdb: Options.table_properties_collectors: 2024-01-24T14:20:34.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: rocksdb: Options.inplace_update_support: 0 2024-01-24T14:20:34.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: rocksdb: Options.inplace_update_num_locks: 10000 2024-01-24T14:20:34.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-24T14:20:34.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-24T14:20:34.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: rocksdb: Options.memtable_huge_page_size: 0 2024-01-24T14:20:34.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: rocksdb: Options.bloom_locality: 0 2024-01-24T14:20:34.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: rocksdb: Options.max_successive_merges: 0 2024-01-24T14:20:34.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: rocksdb: Options.optimize_filters_for_hits: 0 2024-01-24T14:20:34.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: rocksdb: Options.paranoid_file_checks: 0 2024-01-24T14:20:34.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: rocksdb: Options.force_consistency_checks: 1 2024-01-24T14:20:34.846 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: rocksdb: Options.report_bg_io_stats: 0 2024-01-24T14:20:34.846 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: rocksdb: Options.ttl: 2592000 2024-01-24T14:20:34.846 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: rocksdb: Options.periodic_compaction_seconds: 0 2024-01-24T14:20:34.846 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-01-24T14:20:34.846 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-01-24T14:20:34.846 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: rocksdb: Options.enable_blob_files: false 2024-01-24T14:20:34.846 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: rocksdb: Options.min_blob_size: 0 2024-01-24T14:20:34.846 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: rocksdb: Options.blob_file_size: 268435456 2024-01-24T14:20:34.846 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: rocksdb: Options.blob_compression_type: NoCompression 2024-01-24T14:20:34.846 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: rocksdb: Options.enable_blob_garbage_collection: false 2024-01-24T14:20:34.846 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-01-24T14:20:34.846 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-01-24T14:20:34.846 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-01-24T14:20:34.846 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: rocksdb: Options.blob_file_starting_level: 0 2024-01-24T14:20:34.847 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-01-24T14:20:34.847 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-24T14:20:34.847 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-01-24T14:20:34.847 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 478d307e-9845-4ed9-81e6-155315e9cc80 2024-01-24T14:20:34.847 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: rocksdb: EVENT_LOG_v1 {"time_micros": 1706106034590439, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-01-24T14:20:34.847 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-01-24T14:20:34.847 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: rocksdb: EVENT_LOG_v1 {"time_micros": 1706106034590883, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1706106034, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "478d307e-9845-4ed9-81e6-155315e9cc80", "db_session_id": "8HC7A19F82MV0X0NSQQN", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-01-24T14:20:34.847 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: rocksdb: EVENT_LOG_v1 {"time_micros": 1706106034590961, "job": 1, "event": "recovery_finished"} 2024-01-24T14:20:34.847 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-01-24T14:20:34.847 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-01-24T14:20:34.847 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x555dbf93a000 2024-01-24T14:20:34.847 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: rocksdb: DB pointer 0x555dbf928000 2024-01-24T14:20:34.847 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-01-24T14:20:34.847 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-01-24T14:20:34.848 INFO:journalctl@ceph.mon.c.smithi194.stdout: ** DB Stats ** 2024-01-24T14:20:34.848 INFO:journalctl@ceph.mon.c.smithi194.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-01-24T14:20:34.848 INFO:journalctl@ceph.mon.c.smithi194.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-24T14:20:34.848 INFO:journalctl@ceph.mon.c.smithi194.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-24T14:20:34.848 INFO:journalctl@ceph.mon.c.smithi194.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-24T14:20:34.848 INFO:journalctl@ceph.mon.c.smithi194.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-24T14:20:34.848 INFO:journalctl@ceph.mon.c.smithi194.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-24T14:20:34.848 INFO:journalctl@ceph.mon.c.smithi194.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-24T14:20:34.848 INFO:journalctl@ceph.mon.c.smithi194.stdout: 2024-01-24T14:20:34.848 INFO:journalctl@ceph.mon.c.smithi194.stdout: ** Compaction Stats [default] ** 2024-01-24T14:20:34.848 INFO:journalctl@ceph.mon.c.smithi194.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-01-24T14:20:34.848 INFO:journalctl@ceph.mon.c.smithi194.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-01-24T14:20:34.848 INFO:journalctl@ceph.mon.c.smithi194.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-01-24T14:20:34.848 INFO:journalctl@ceph.mon.c.smithi194.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-01-24T14:20:34.849 INFO:journalctl@ceph.mon.c.smithi194.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-01-24T14:20:34.849 INFO:journalctl@ceph.mon.c.smithi194.stdout: 2024-01-24T14:20:34.849 INFO:journalctl@ceph.mon.c.smithi194.stdout: ** Compaction Stats [default] ** 2024-01-24T14:20:34.849 INFO:journalctl@ceph.mon.c.smithi194.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-01-24T14:20:34.849 INFO:journalctl@ceph.mon.c.smithi194.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-24T14:20:34.849 INFO:journalctl@ceph.mon.c.smithi194.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-01-24T14:20:34.849 INFO:journalctl@ceph.mon.c.smithi194.stdout: 2024-01-24T14:20:34.849 INFO:journalctl@ceph.mon.c.smithi194.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-01-24T14:20:34.849 INFO:journalctl@ceph.mon.c.smithi194.stdout: 2024-01-24T14:20:34.849 INFO:journalctl@ceph.mon.c.smithi194.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-01-24T14:20:34.849 INFO:journalctl@ceph.mon.c.smithi194.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-01-24T14:20:34.849 INFO:journalctl@ceph.mon.c.smithi194.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-24T14:20:34.849 INFO:journalctl@ceph.mon.c.smithi194.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-01-24T14:20:34.849 INFO:journalctl@ceph.mon.c.smithi194.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-01-24T14:20:34.849 INFO:journalctl@ceph.mon.c.smithi194.stdout: AddFile(Keys): cumulative 0, interval 0 2024-01-24T14:20:34.850 INFO:journalctl@ceph.mon.c.smithi194.stdout: Cumulative compaction: 0.00 GB write, 0.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-24T14:20:34.850 INFO:journalctl@ceph.mon.c.smithi194.stdout: Interval compaction: 0.00 GB write, 0.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-24T14:20:34.850 INFO:journalctl@ceph.mon.c.smithi194.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-24T14:20:34.850 INFO:journalctl@ceph.mon.c.smithi194.stdout: Block cache BinnedLRUCache@0x555dbf89d1f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-01-24T14:20:34.850 INFO:journalctl@ceph.mon.c.smithi194.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-01-24T14:20:34.850 INFO:journalctl@ceph.mon.c.smithi194.stdout: 2024-01-24T14:20:34.850 INFO:journalctl@ceph.mon.c.smithi194.stdout: ** File Read Latency Histogram By Level [default] ** 2024-01-24T14:20:34.850 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-01-24T14:20:34.850 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: using public_addr v2:172.21.15.194:0/0 -> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-01-24T14:20:34.850 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: starting mon.c rank -1 at public addrs [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] at bind addrs [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 2024-01-24T14:20:34.850 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: mon.c@-1(???) e0 preinit fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 2024-01-24T14:20:34.850 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: mon.c@-1(synchronizing).mds e1 new map 2024-01-24T14:20:34.851 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: mon.c@-1(synchronizing).mds e1 print_map 2024-01-24T14:20:34.851 INFO:journalctl@ceph.mon.c.smithi194.stdout: e1 2024-01-24T14:20:34.851 INFO:journalctl@ceph.mon.c.smithi194.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-01-24T14:20:34.851 INFO:journalctl@ceph.mon.c.smithi194.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-24T14:20:34.851 INFO:journalctl@ceph.mon.c.smithi194.stdout: legacy client fscid: -1 2024-01-24T14:20:34.851 INFO:journalctl@ceph.mon.c.smithi194.stdout: 2024-01-24T14:20:34.851 INFO:journalctl@ceph.mon.c.smithi194.stdout: No filesystems configured 2024-01-24T14:20:34.851 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-01-24T14:20:34.851 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-01-24T14:20:34.851 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-01-24T14:20:34.851 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-01-24T14:20:34.851 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-01-24T14:20:34.851 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-01-24T14:20:34.851 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-01-24T14:20:34.851 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-01-24T14:20:34.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-01-24T14:20:34.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-01-24T14:20:34.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:34.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-24T14:20:34.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:34.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:34.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:34.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:34.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi194", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:20:34.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:20:34.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:20:34.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: Updating smithi194:/etc/ceph/ceph.conf 2024-01-24T14:20:34.852 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: Updating smithi194:/var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/config/ceph.conf 2024-01-24T14:20:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: Updating smithi194:/etc/ceph/ceph.client.admin.keyring 2024-01-24T14:20:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/932515429' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-01-24T14:20:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: Updating smithi194:/var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/config/ceph.client.admin.keyring 2024-01-24T14:20:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/932515429' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-01-24T14:20:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: osdmap e4: 0 total, 0 up, 0 in 2024-01-24T14:20:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi019:172.21.15.19=a;smithi123:172.21.15.123=b;smithi194:172.21.15.194=c", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:20:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: Saving service mon spec with placement smithi019:172.21.15.19=a;smithi123:172.21.15.123=b;smithi194:172.21.15.194=c;count:3 2024-01-24T14:20:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:34.853 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:20:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:20:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:20:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-24T14:20:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:20:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: Deploying daemon mon.c on smithi194 2024-01-24T14:20:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: from='client.? 172.21.15.194:0/1173817460' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-24T14:20:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-01-24T14:20:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2024-01-24T14:20:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: expand_channel_meta expand map: {default=false} 2024-01-24T14:20:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: expand_channel_meta from 'false' to 'false' 2024-01-24T14:20:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: expand_channel_meta expanded map: {default=false} 2024-01-24T14:20:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: expand_channel_meta expand map: {default=info} 2024-01-24T14:20:34.854 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: expand_channel_meta from 'info' to 'info' 2024-01-24T14:20:34.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: expand_channel_meta expanded map: {default=info} 2024-01-24T14:20:34.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: expand_channel_meta expand map: {default=daemon} 2024-01-24T14:20:34.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: expand_channel_meta from 'daemon' to 'daemon' 2024-01-24T14:20:34.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: expand_channel_meta expanded map: {default=daemon} 2024-01-24T14:20:34.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: expand_channel_meta expand map: {default=debug} 2024-01-24T14:20:34.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: expand_channel_meta from 'debug' to 'debug' 2024-01-24T14:20:34.855 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:34 smithi194 ceph-mon[45450]: expand_channel_meta expanded map: {default=debug} 2024-01-24T14:20:35.380 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-01-24T14:20:35.380 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 -- ceph mon dump -f json 2024-01-24T14:20:35.599 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/mon.c/config 2024-01-24T14:20:39.757 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:39.754+0000 7fc98a620700 1 -- 172.21.15.194:0/3756899541 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc96c0035c0 msgr2=0x7fc96c005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:39.757 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:39.754+0000 7fc98a620700 1 --2- 172.21.15.194:0/3756899541 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc96c0035c0 0x7fc96c005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc984068d70 tx=0x7fc974013e90 comp rx=0 tx=0).stop 2024-01-24T14:20:39.757 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:39.754+0000 7fc98a620700 1 -- 172.21.15.194:0/3756899541 shutdown_connections 2024-01-24T14:20:39.757 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:39.754+0000 7fc98a620700 1 --2- 172.21.15.194:0/3756899541 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc96c0035c0 0x7fc96c005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:39.757 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:39.754+0000 7fc98a620700 1 -- 172.21.15.194:0/3756899541 >> 172.21.15.194:0/3756899541 conn(0x7fc9840fb300 msgr2=0x7fc9840fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:20:39.757 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:39.755+0000 7fc98a620700 1 -- 172.21.15.194:0/3756899541 shutdown_connections 2024-01-24T14:20:39.757 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:39.755+0000 7fc98a620700 1 -- 172.21.15.194:0/3756899541 wait complete. 2024-01-24T14:20:39.758 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:39.755+0000 7fc98a620700 1 Processor -- start 2024-01-24T14:20:39.758 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:39.755+0000 7fc98a620700 1 -- start start 2024-01-24T14:20:39.758 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:39.756+0000 7fc98a620700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc96c0035c0 0x7fc9840ff7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:39.758 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:39.756+0000 7fc98a620700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc9840ffcf0 0x7fc984100170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:39.758 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:39.756+0000 7fc98a620700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc984101730 con 0x7fc96c0035c0 2024-01-24T14:20:39.758 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:39.756+0000 7fc98a620700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9841a1f50 con 0x7fc9840ffcf0 2024-01-24T14:20:39.759 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:39.756+0000 7fc9837fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc9840ffcf0 0x7fc984100170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:39.759 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:39.756+0000 7fc9837fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc9840ffcf0 0x7fc984100170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:37688/0 (socket says 172.21.15.194:37688) 2024-01-24T14:20:39.759 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:39.756+0000 7fc9837fe700 1 -- 172.21.15.194:0/1712827832 learned_addr learned my addr 172.21.15.194:0/1712827832 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-24T14:20:39.760 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:39.756+0000 7fc9837fe700 1 -- 172.21.15.194:0/1712827832 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc9840ffcf0 msgr2=0x7fc984100170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-01-24T14:20:39.760 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:39.756+0000 7fc9837fe700 1 -- 172.21.15.194:0/1712827832 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc9840ffcf0 msgr2=0x7fc984100170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-01-24T14:20:39.760 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:39.756+0000 7fc9837fe700 1 --2- 172.21.15.194:0/1712827832 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc9840ffcf0 0x7fc984100170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-24T14:20:39.760 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:39.756+0000 7fc9837fe700 1 --2- 172.21.15.194:0/1712827832 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc9840ffcf0 0x7fc984100170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-24T14:20:39.760 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:39.756+0000 7fc983fff700 1 --2- 172.21.15.194:0/1712827832 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc96c0035c0 0x7fc9840ff7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:39.760 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:39.757+0000 7fc983fff700 1 -- 172.21.15.194:0/1712827832 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc9840ffcf0 msgr2=0x7fc984100170 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-24T14:20:39.760 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:39.757+0000 7fc983fff700 1 --2- 172.21.15.194:0/1712827832 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc9840ffcf0 0x7fc984100170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:39.760 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:39.757+0000 7fc983fff700 1 -- 172.21.15.194:0/1712827832 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9740138e0 con 0x7fc96c0035c0 2024-01-24T14:20:39.760 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:39.757+0000 7fc983fff700 1 --2- 172.21.15.194:0/1712827832 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc96c0035c0 0x7fc9840ff7b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc97401c000 tx=0x7fc974005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:20:39.760 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:39.757+0000 7fc9817fa700 1 -- 172.21.15.194:0/1712827832 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fc974018070 con 0x7fc96c0035c0 2024-01-24T14:20:39.760 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:39.757+0000 7fc98a620700 1 -- 172.21.15.194:0/1712827832 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9841a2150 con 0x7fc96c0035c0 2024-01-24T14:20:39.760 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:39.757+0000 7fc9817fa700 1 -- 172.21.15.194:0/1712827832 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc97400feb0 con 0x7fc96c0035c0 2024-01-24T14:20:39.760 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:39.757+0000 7fc9817fa700 1 -- 172.21.15.194:0/1712827832 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fc974003d70 con 0x7fc96c0035c0 2024-01-24T14:20:39.761 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:39.758+0000 7fc98a620700 1 -- 172.21.15.194:0/1712827832 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9841a2670 con 0x7fc96c0035c0 2024-01-24T14:20:39.761 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:39.758+0000 7fc9817fa700 1 -- 172.21.15.194:0/1712827832 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 13) v1 ==== 45324+0+0 (secure 0 0 0) 0x7fc97401c410 con 0x7fc96c0035c0 2024-01-24T14:20:39.761 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:39.759+0000 7fc9817fa700 1 --2- 172.21.15.194:0/1712827832 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fc9700381e0 0x7fc97003a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:39.761 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:39.759+0000 7fc9817fa700 1 -- 172.21.15.194:0/1712827832 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc97404eaf0 con 0x7fc96c0035c0 2024-01-24T14:20:39.761 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:39.759+0000 7fc98a620700 1 -- 172.21.15.194:0/1712827832 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc984062370 con 0x7fc96c0035c0 2024-01-24T14:20:39.762 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:39.759+0000 7fc9837fe700 1 --2- 172.21.15.194:0/1712827832 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fc9700381e0 0x7fc97003a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:39.762 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:39.760+0000 7fc9837fe700 1 --2- 172.21.15.194:0/1712827832 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fc9700381e0 0x7fc97003a6a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc978009180 tx=0x7fc978008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:20:39.766 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:39.765+0000 7fc9817fa700 1 -- 172.21.15.194:0/1712827832 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fc974025400 con 0x7fc96c0035c0 2024-01-24T14:20:39.985 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:39 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-01-24T14:20:39.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:39 smithi019 ceph-mon[40663]: mon.a calling monitor election 2024-01-24T14:20:39.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:39 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T14:20:39.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:39 smithi019 ceph-mon[40663]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:39.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:39 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T14:20:39.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:39 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T14:20:39.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:39 smithi019 ceph-mon[40663]: mon.c calling monitor election 2024-01-24T14:20:39.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:39 smithi019 ceph-mon[40663]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:39.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:39 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T14:20:39.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:39 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T14:20:39.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:39 smithi019 ceph-mon[40663]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:39.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:39 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T14:20:39.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:39 smithi019 ceph-mon[40663]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-01-24T14:20:39.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:39 smithi019 ceph-mon[40663]: monmap e2: 2 mons at {a=[v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0],c=[v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-01-24T14:20:39.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:39 smithi019 ceph-mon[40663]: fsmap 2024-01-24T14:20:39.987 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:39 smithi019 ceph-mon[40663]: osdmap e4: 0 total, 0 up, 0 in 2024-01-24T14:20:39.987 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:39 smithi019 ceph-mon[40663]: mgrmap e13: x(active, since 36s) 2024-01-24T14:20:39.987 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:39 smithi019 ceph-mon[40663]: overall HEALTH_OK 2024-01-24T14:20:39.987 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:39 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:39.987 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:39 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:39.987 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:39 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:39.987 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:39 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-24T14:20:39.987 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:39 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:20:40.031 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:40.030+0000 7fc98a620700 1 -- 172.21.15.194:0/1712827832 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc9841a28c0 con 0x7fc96c0035c0 2024-01-24T14:20:40.032 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:40.031+0000 7fc9817fa700 1 -- 172.21.15.194:0/1712827832 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1006 (secure 0 0 0) 0x7fc97403c090 con 0x7fc96c0035c0 2024-01-24T14:20:40.032 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-24T14:20:40.032 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":2,"fsid":"654b0c8e-bac3-11ee-95b1-87774f69a715","modified":"2024-01-24T14:20:34.619437Z","created":"2024-01-24T14:19:08.559591Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-01-24T14:20:40.034 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:40.033+0000 7fc98a620700 1 -- 172.21.15.194:0/1712827832 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fc9700381e0 msgr2=0x7fc97003a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:40.034 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:40.033+0000 7fc98a620700 1 --2- 172.21.15.194:0/1712827832 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fc9700381e0 0x7fc97003a6a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc978009180 tx=0x7fc978008040 comp rx=0 tx=0).stop 2024-01-24T14:20:40.034 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:40.033+0000 7fc98a620700 1 -- 172.21.15.194:0/1712827832 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc96c0035c0 msgr2=0x7fc9840ff7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:40.034 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:40.033+0000 7fc98a620700 1 --2- 172.21.15.194:0/1712827832 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc96c0035c0 0x7fc9840ff7b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc97401c000 tx=0x7fc974005050 comp rx=0 tx=0).stop 2024-01-24T14:20:40.034 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:40.033+0000 7fc98a620700 1 -- 172.21.15.194:0/1712827832 shutdown_connections 2024-01-24T14:20:40.034 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:40.033+0000 7fc98a620700 1 --2- 172.21.15.194:0/1712827832 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fc9700381e0 0x7fc97003a6a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:40.034 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:40.033+0000 7fc98a620700 1 --2- 172.21.15.194:0/1712827832 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc96c0035c0 0x7fc9840ff7b0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:40.034 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:40.033+0000 7fc98a620700 1 --2- 172.21.15.194:0/1712827832 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc9840ffcf0 0x7fc984100170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:40.034 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:40.033+0000 7fc98a620700 1 -- 172.21.15.194:0/1712827832 >> 172.21.15.194:0/1712827832 conn(0x7fc9840fb300 msgr2=0x7fc98418e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:20:40.035 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:40.033+0000 7fc98a620700 1 -- 172.21.15.194:0/1712827832 shutdown_connections 2024-01-24T14:20:40.035 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:40.033+0000 7fc98a620700 1 -- 172.21.15.194:0/1712827832 wait complete. 2024-01-24T14:20:40.035 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 2 2024-01-24T14:20:40.058 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:39 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-01-24T14:20:40.058 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:39 smithi194 ceph-mon[45450]: mon.a calling monitor election 2024-01-24T14:20:40.058 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:39 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T14:20:40.058 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:39 smithi194 ceph-mon[45450]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:40.058 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:39 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T14:20:40.058 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:39 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T14:20:40.058 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:39 smithi194 ceph-mon[45450]: mon.c calling monitor election 2024-01-24T14:20:40.058 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:39 smithi194 ceph-mon[45450]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:40.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:39 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T14:20:40.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:39 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T14:20:40.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:39 smithi194 ceph-mon[45450]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:40.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:39 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T14:20:40.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:39 smithi194 ceph-mon[45450]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-01-24T14:20:40.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:39 smithi194 ceph-mon[45450]: monmap e2: 2 mons at {a=[v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0],c=[v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-01-24T14:20:40.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:39 smithi194 ceph-mon[45450]: fsmap 2024-01-24T14:20:40.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:39 smithi194 ceph-mon[45450]: osdmap e4: 0 total, 0 up, 0 in 2024-01-24T14:20:40.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:39 smithi194 ceph-mon[45450]: mgrmap e13: x(active, since 36s) 2024-01-24T14:20:40.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:39 smithi194 ceph-mon[45450]: overall HEALTH_OK 2024-01-24T14:20:40.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:39 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:40.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:39 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:40.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:39 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:40.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:39 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-24T14:20:40.059 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:39 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:20:41.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:40 smithi194 ceph-mon[45450]: Deploying daemon mon.b on smithi123 2024-01-24T14:20:41.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:40 smithi194 ceph-mon[45450]: from='client.? 172.21.15.194:0/1712827832' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-24T14:20:41.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:40 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T14:20:41.235 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:40 smithi019 ceph-mon[40663]: Deploying daemon mon.b on smithi123 2024-01-24T14:20:41.235 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:40 smithi019 ceph-mon[40663]: from='client.? 172.21.15.194:0/1712827832' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-24T14:20:41.235 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:40 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T14:20:41.409 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-01-24T14:20:41.409 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 -- ceph mon dump -f json 2024-01-24T14:20:41.638 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/mon.c/config 2024-01-24T14:20:42.003 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:41 smithi194 ceph-mon[45450]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:42.235 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:41 smithi019 ceph-mon[40663]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:42.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 systemd[1]: Started Ceph mon.b for 654b0c8e-bac3-11ee-95b1-87774f69a715. 2024-01-24T14:20:42.551 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: rocksdb: EVENT_LOG_v1 {"time_micros": 1706106042285998, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1706106042, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "16ede44e-7226-4c3e-bfda-b793d9c20d91", "db_session_id": "C32DP6WJFJ1XG8BO2KA4", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-01-24T14:20:42.551 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: rocksdb: EVENT_LOG_v1 {"time_micros": 1706106042286087, "job": 1, "event": "recovery_finished"} 2024-01-24T14:20:42.551 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-01-24T14:20:42.551 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-01-24T14:20:42.551 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5651835e2000 2024-01-24T14:20:42.551 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: rocksdb: DB pointer 0x5651835d0000 2024-01-24T14:20:42.551 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-01-24T14:20:42.551 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-01-24T14:20:42.551 INFO:journalctl@ceph.mon.b.smithi123.stdout: ** DB Stats ** 2024-01-24T14:20:42.551 INFO:journalctl@ceph.mon.b.smithi123.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-01-24T14:20:42.551 INFO:journalctl@ceph.mon.b.smithi123.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-24T14:20:42.552 INFO:journalctl@ceph.mon.b.smithi123.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-24T14:20:42.552 INFO:journalctl@ceph.mon.b.smithi123.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-24T14:20:42.552 INFO:journalctl@ceph.mon.b.smithi123.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-24T14:20:42.552 INFO:journalctl@ceph.mon.b.smithi123.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-24T14:20:42.552 INFO:journalctl@ceph.mon.b.smithi123.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-24T14:20:42.552 INFO:journalctl@ceph.mon.b.smithi123.stdout: 2024-01-24T14:20:42.552 INFO:journalctl@ceph.mon.b.smithi123.stdout: ** Compaction Stats [default] ** 2024-01-24T14:20:42.552 INFO:journalctl@ceph.mon.b.smithi123.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-01-24T14:20:42.552 INFO:journalctl@ceph.mon.b.smithi123.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-01-24T14:20:42.552 INFO:journalctl@ceph.mon.b.smithi123.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-01-24T14:20:42.552 INFO:journalctl@ceph.mon.b.smithi123.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-01-24T14:20:42.552 INFO:journalctl@ceph.mon.b.smithi123.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-01-24T14:20:42.553 INFO:journalctl@ceph.mon.b.smithi123.stdout: 2024-01-24T14:20:42.553 INFO:journalctl@ceph.mon.b.smithi123.stdout: ** Compaction Stats [default] ** 2024-01-24T14:20:42.553 INFO:journalctl@ceph.mon.b.smithi123.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-01-24T14:20:42.553 INFO:journalctl@ceph.mon.b.smithi123.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-24T14:20:42.553 INFO:journalctl@ceph.mon.b.smithi123.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-01-24T14:20:42.553 INFO:journalctl@ceph.mon.b.smithi123.stdout: 2024-01-24T14:20:42.553 INFO:journalctl@ceph.mon.b.smithi123.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-01-24T14:20:42.553 INFO:journalctl@ceph.mon.b.smithi123.stdout: 2024-01-24T14:20:42.553 INFO:journalctl@ceph.mon.b.smithi123.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-01-24T14:20:42.553 INFO:journalctl@ceph.mon.b.smithi123.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-01-24T14:20:42.553 INFO:journalctl@ceph.mon.b.smithi123.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-24T14:20:42.553 INFO:journalctl@ceph.mon.b.smithi123.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-01-24T14:20:42.554 INFO:journalctl@ceph.mon.b.smithi123.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-01-24T14:20:42.554 INFO:journalctl@ceph.mon.b.smithi123.stdout: AddFile(Keys): cumulative 0, interval 0 2024-01-24T14:20:42.555 INFO:journalctl@ceph.mon.b.smithi123.stdout: Cumulative compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-24T14:20:42.555 INFO:journalctl@ceph.mon.b.smithi123.stdout: Interval compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-24T14:20:42.555 INFO:journalctl@ceph.mon.b.smithi123.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-24T14:20:42.555 INFO:journalctl@ceph.mon.b.smithi123.stdout: Block cache BinnedLRUCache@0x5651835451f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-01-24T14:20:42.555 INFO:journalctl@ceph.mon.b.smithi123.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-01-24T14:20:42.555 INFO:journalctl@ceph.mon.b.smithi123.stdout: 2024-01-24T14:20:42.555 INFO:journalctl@ceph.mon.b.smithi123.stdout: ** File Read Latency Histogram By Level [default] ** 2024-01-24T14:20:42.555 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-01-24T14:20:42.555 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: using public_addr v2:172.21.15.123:0/0 -> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] 2024-01-24T14:20:42.555 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: starting mon.b rank -1 at public addrs [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] at bind addrs [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 2024-01-24T14:20:42.555 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: mon.b@-1(???) e0 preinit fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 2024-01-24T14:20:42.555 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: mon.b@-1(synchronizing).mds e1 new map 2024-01-24T14:20:42.555 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: mon.b@-1(synchronizing).mds e1 print_map 2024-01-24T14:20:42.555 INFO:journalctl@ceph.mon.b.smithi123.stdout: e1 2024-01-24T14:20:42.556 INFO:journalctl@ceph.mon.b.smithi123.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-01-24T14:20:42.556 INFO:journalctl@ceph.mon.b.smithi123.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-24T14:20:42.556 INFO:journalctl@ceph.mon.b.smithi123.stdout: legacy client fscid: -1 2024-01-24T14:20:42.556 INFO:journalctl@ceph.mon.b.smithi123.stdout: 2024-01-24T14:20:42.556 INFO:journalctl@ceph.mon.b.smithi123.stdout: No filesystems configured 2024-01-24T14:20:42.556 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-01-24T14:20:42.556 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-01-24T14:20:42.556 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-01-24T14:20:42.556 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-01-24T14:20:42.556 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-01-24T14:20:42.556 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-01-24T14:20:42.556 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-01-24T14:20:42.557 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-01-24T14:20:42.557 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-01-24T14:20:42.557 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-01-24T14:20:42.557 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:42.557 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-24T14:20:42.557 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:42.557 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:42.557 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:42.557 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:42.557 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi194", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:20:42.557 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:20:42.557 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:20:42.557 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: Updating smithi194:/etc/ceph/ceph.conf 2024-01-24T14:20:42.557 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: Updating smithi194:/var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/config/ceph.conf 2024-01-24T14:20:42.558 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:42.558 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: Updating smithi194:/etc/ceph/ceph.client.admin.keyring 2024-01-24T14:20:42.558 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/932515429' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-01-24T14:20:42.558 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: Updating smithi194:/var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/config/ceph.client.admin.keyring 2024-01-24T14:20:42.558 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:42.558 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:42.558 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:42.558 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/932515429' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-01-24T14:20:42.558 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: osdmap e4: 0 total, 0 up, 0 in 2024-01-24T14:20:42.559 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:42.559 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi019:172.21.15.19=a;smithi123:172.21.15.123=b;smithi194:172.21.15.194=c", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:20:42.559 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: Saving service mon spec with placement smithi019:172.21.15.19=a;smithi123:172.21.15.123=b;smithi194:172.21.15.194=c;count:3 2024-01-24T14:20:42.559 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:42.559 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:20:42.559 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:20:42.559 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:20:42.559 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:42.559 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-24T14:20:42.559 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:20:42.559 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: Deploying daemon mon.c on smithi194 2024-01-24T14:20:42.559 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:42.559 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: from='client.? 172.21.15.194:0/1173817460' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-24T14:20:42.559 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-01-24T14:20:42.559 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: mon.a calling monitor election 2024-01-24T14:20:42.560 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T14:20:42.560 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:42.560 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T14:20:42.560 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T14:20:42.560 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: mon.c calling monitor election 2024-01-24T14:20:42.560 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:42.560 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T14:20:42.560 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T14:20:42.560 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:42.560 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T14:20:42.560 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-01-24T14:20:42.560 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: monmap e2: 2 mons at {a=[v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0],c=[v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-01-24T14:20:42.560 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: fsmap 2024-01-24T14:20:42.560 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: osdmap e4: 0 total, 0 up, 0 in 2024-01-24T14:20:42.561 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: mgrmap e13: x(active, since 36s) 2024-01-24T14:20:42.561 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: overall HEALTH_OK 2024-01-24T14:20:42.561 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:42.561 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:42.561 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:42.561 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-24T14:20:42.561 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:20:42.561 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: Deploying daemon mon.b on smithi123 2024-01-24T14:20:42.561 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: from='client.? 172.21.15.194:0/1712827832' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-24T14:20:42.561 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T14:20:42.561 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:42.561 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-01-24T14:20:42.561 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete 2024-01-24T14:20:42.561 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: expand_channel_meta expand map: {default=false} 2024-01-24T14:20:42.562 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: expand_channel_meta from 'false' to 'false' 2024-01-24T14:20:42.562 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: expand_channel_meta expanded map: {default=false} 2024-01-24T14:20:42.562 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: expand_channel_meta expand map: {default=info} 2024-01-24T14:20:42.562 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: expand_channel_meta from 'info' to 'info' 2024-01-24T14:20:42.562 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: expand_channel_meta expanded map: {default=info} 2024-01-24T14:20:42.562 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: expand_channel_meta expand map: {default=daemon} 2024-01-24T14:20:42.562 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: expand_channel_meta from 'daemon' to 'daemon' 2024-01-24T14:20:42.562 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: expand_channel_meta expanded map: {default=daemon} 2024-01-24T14:20:42.562 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: expand_channel_meta expand map: {default=debug} 2024-01-24T14:20:42.562 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: expand_channel_meta from 'debug' to 'debug' 2024-01-24T14:20:42.562 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:42 smithi123 ceph-mon[44966]: expand_channel_meta expanded map: {default=debug} 2024-01-24T14:20:47.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:47 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-01-24T14:20:47.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:47 smithi019 ceph-mon[40663]: mon.a calling monitor election 2024-01-24T14:20:47.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:47 smithi019 ceph-mon[40663]: mon.c calling monitor election 2024-01-24T14:20:47.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:47 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T14:20:47.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:47 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T14:20:47.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:47 smithi019 ceph-mon[40663]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:47.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:47 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T14:20:47.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:47 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T14:20:47.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:47 smithi019 ceph-mon[40663]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:47.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:47 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T14:20:47.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:47 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T14:20:47.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:47 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T14:20:47.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:47 smithi019 ceph-mon[40663]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-01-24T14:20:47.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:47 smithi019 ceph-mon[40663]: monmap e3: 3 mons at {a=[v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0],b=[v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0],c=[v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-01-24T14:20:47.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:47 smithi019 ceph-mon[40663]: fsmap 2024-01-24T14:20:47.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:47 smithi019 ceph-mon[40663]: osdmap e4: 0 total, 0 up, 0 in 2024-01-24T14:20:47.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:47 smithi019 ceph-mon[40663]: mgrmap e13: x(active, since 44s) 2024-01-24T14:20:47.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:47 smithi019 ceph-mon[40663]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-01-24T14:20:47.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:47 smithi019 ceph-mon[40663]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-01-24T14:20:47.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:47 smithi019 ceph-mon[40663]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-01-24T14:20:47.738 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:47 smithi019 ceph-mon[40663]: mon.b (rank 2) addr [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] is down (out of quorum) 2024-01-24T14:20:47.738 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:47 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:47.738 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:47 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:47.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:47 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-01-24T14:20:47.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:47 smithi194 ceph-mon[45450]: mon.a calling monitor election 2024-01-24T14:20:47.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:47 smithi194 ceph-mon[45450]: mon.c calling monitor election 2024-01-24T14:20:47.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:47 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T14:20:47.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:47 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T14:20:47.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:47 smithi194 ceph-mon[45450]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:47.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:47 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T14:20:47.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:47 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T14:20:47.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:47 smithi194 ceph-mon[45450]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:47.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:47 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T14:20:47.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:47 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T14:20:47.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:47 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T14:20:47.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:47 smithi194 ceph-mon[45450]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-01-24T14:20:47.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:47 smithi194 ceph-mon[45450]: monmap e3: 3 mons at {a=[v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0],b=[v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0],c=[v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-01-24T14:20:47.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:47 smithi194 ceph-mon[45450]: fsmap 2024-01-24T14:20:47.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:47 smithi194 ceph-mon[45450]: osdmap e4: 0 total, 0 up, 0 in 2024-01-24T14:20:47.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:47 smithi194 ceph-mon[45450]: mgrmap e13: x(active, since 44s) 2024-01-24T14:20:47.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:47 smithi194 ceph-mon[45450]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-01-24T14:20:47.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:47 smithi194 ceph-mon[45450]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-01-24T14:20:47.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:47 smithi194 ceph-mon[45450]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-01-24T14:20:47.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:47 smithi194 ceph-mon[45450]: mon.b (rank 2) addr [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] is down (out of quorum) 2024-01-24T14:20:47.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:47 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:47.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:47 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:48.549 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.547+0000 7f6300c1a700 1 -- 172.21.15.194:0/2625491709 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62e4006370 msgr2=0x7f62e40087e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:48.549 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.547+0000 7f6300c1a700 1 --2- 172.21.15.194:0/2625491709 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62e4006370 0x7f62e40087e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f62fc068ef0 tx=0x7f62ec00fe90 comp rx=0 tx=0).stop 2024-01-24T14:20:48.549 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.547+0000 7f6300c1a700 1 -- 172.21.15.194:0/2625491709 shutdown_connections 2024-01-24T14:20:48.550 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.547+0000 7f6300c1a700 1 --2- 172.21.15.194:0/2625491709 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62e4006370 0x7f62e40087e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:48.550 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.547+0000 7f6300c1a700 1 -- 172.21.15.194:0/2625491709 >> 172.21.15.194:0/2625491709 conn(0x7f62fc0fb300 msgr2=0x7f62fc0fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:20:48.550 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.547+0000 7f6300c1a700 1 -- 172.21.15.194:0/2625491709 shutdown_connections 2024-01-24T14:20:48.550 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.547+0000 7f6300c1a700 1 -- 172.21.15.194:0/2625491709 wait complete. 2024-01-24T14:20:48.550 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.548+0000 7f6300c1a700 1 Processor -- start 2024-01-24T14:20:48.550 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.548+0000 7f6300c1a700 1 -- start start 2024-01-24T14:20:48.550 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.548+0000 7f6300c1a700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f62e4006370 0x7f62fc195e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:48.550 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.548+0000 7f6300c1a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62fc1963d0 0x7f62fc19a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:48.550 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.548+0000 7f6300c1a700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62fc19ad80 0x7f62fc19b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:48.550 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.548+0000 7f6300c1a700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62fc196ac0 con 0x7f62fc19ad80 2024-01-24T14:20:48.550 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.548+0000 7f6300c1a700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62fc1a2af0 con 0x7f62fc1963d0 2024-01-24T14:20:48.551 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.548+0000 7f6300c1a700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62fc1a2c30 con 0x7f62e4006370 2024-01-24T14:20:48.551 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.548+0000 7f62fa59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f62e4006370 0x7f62fc195e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:48.551 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.548+0000 7f62fa59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f62e4006370 0x7f62fc195e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:44784/0 (socket says 172.21.15.194:44784) 2024-01-24T14:20:48.551 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.548+0000 7f62fa59c700 1 -- 172.21.15.194:0/259542543 learned_addr learned my addr 172.21.15.194:0/259542543 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-24T14:20:48.551 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.548+0000 7f62fad9d700 1 --2- 172.21.15.194:0/259542543 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62fc19ad80 0x7f62fc19b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:48.551 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.548+0000 7f62fa59c700 1 -- 172.21.15.194:0/259542543 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f62e4006370 msgr2=0x7f62fc195e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-01-24T14:20:48.551 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.548+0000 7f62fa59c700 1 -- 172.21.15.194:0/259542543 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f62e4006370 msgr2=0x7f62fc195e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-01-24T14:20:48.551 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.548+0000 7f62fa59c700 1 --2- 172.21.15.194:0/259542543 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f62e4006370 0x7f62fc195e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-24T14:20:48.551 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.548+0000 7f62f9d9b700 1 --2- 172.21.15.194:0/259542543 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62fc1963d0 0x7f62fc19a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:48.551 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.548+0000 7f62fa59c700 1 --2- 172.21.15.194:0/259542543 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f62e4006370 0x7f62fc195e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-24T14:20:48.551 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.548+0000 7f62f9d9b700 1 -- 172.21.15.194:0/259542543 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62fc1963d0 msgr2=0x7f62fc19a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-01-24T14:20:48.552 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.548+0000 7f62f9d9b700 1 -- 172.21.15.194:0/259542543 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62fc1963d0 msgr2=0x7f62fc19a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-01-24T14:20:48.552 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.548+0000 7f62f9d9b700 1 --2- 172.21.15.194:0/259542543 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62fc1963d0 0x7f62fc19a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-24T14:20:48.552 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.548+0000 7f62f9d9b700 1 --2- 172.21.15.194:0/259542543 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62fc1963d0 0x7f62fc19a840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-24T14:20:48.552 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.548+0000 7f62fad9d700 1 -- 172.21.15.194:0/259542543 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f62e4006370 msgr2=0x7f62fc195e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-24T14:20:48.552 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.548+0000 7f62fad9d700 1 --2- 172.21.15.194:0/259542543 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f62e4006370 0x7f62fc195e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:48.552 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.548+0000 7f62fad9d700 1 -- 172.21.15.194:0/259542543 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62fc1963d0 msgr2=0x7f62fc19a840 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-24T14:20:48.552 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.548+0000 7f62fad9d700 1 --2- 172.21.15.194:0/259542543 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62fc1963d0 0x7f62fc19a840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:48.552 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.548+0000 7f62fad9d700 1 -- 172.21.15.194:0/259542543 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62ec00f8e0 con 0x7f62fc19ad80 2024-01-24T14:20:48.552 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.549+0000 7f62fad9d700 1 --2- 172.21.15.194:0/259542543 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62fc19ad80 0x7f62fc19b230 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f62f400bd50 tx=0x7f62f400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:20:48.552 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.549+0000 7f62eb7fe700 1 -- 172.21.15.194:0/259542543 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f400cd10 con 0x7f62fc19ad80 2024-01-24T14:20:48.552 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.549+0000 7f6300c1a700 1 -- 172.21.15.194:0/259542543 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62fc1a2f10 con 0x7f62fc19ad80 2024-01-24T14:20:48.552 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.549+0000 7f62eb7fe700 1 -- 172.21.15.194:0/259542543 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f62f400ce70 con 0x7f62fc19ad80 2024-01-24T14:20:48.553 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.549+0000 7f62eb7fe700 1 -- 172.21.15.194:0/259542543 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f4012840 con 0x7f62fc19ad80 2024-01-24T14:20:48.553 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.549+0000 7f6300c1a700 1 -- 172.21.15.194:0/259542543 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62fc1a3360 con 0x7f62fc19ad80 2024-01-24T14:20:48.553 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.550+0000 7f6300c1a700 1 -- 172.21.15.194:0/259542543 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62fc062370 con 0x7f62fc19ad80 2024-01-24T14:20:48.554 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.550+0000 7f62eb7fe700 1 -- 172.21.15.194:0/259542543 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 13) v1 ==== 45324+0+0 (secure 0 0 0) 0x7f62f4014b00 con 0x7f62fc19ad80 2024-01-24T14:20:48.554 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.550+0000 7f62eb7fe700 1 --2- 172.21.15.194:0/259542543 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f62dc038870 0x7f62dc03ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:48.554 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.550+0000 7f62eb7fe700 1 -- 172.21.15.194:0/259542543 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f62f404a8e0 con 0x7f62fc19ad80 2024-01-24T14:20:48.554 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.550+0000 7f62fa59c700 1 --2- 172.21.15.194:0/259542543 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f62dc038870 0x7f62dc03ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:48.554 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.551+0000 7f62fa59c700 1 --2- 172.21.15.194:0/259542543 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f62dc038870 0x7f62dc03ad30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f62ec005ed0 tx=0x7f62ec005e60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:20:48.556 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.556+0000 7f62eb7fe700 1 -- 172.21.15.194:0/259542543 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f62f401b8d0 con 0x7f62fc19ad80 2024-01-24T14:20:48.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:48 smithi019 ceph-mon[40663]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:48.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:48 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:48.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:48 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:48.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:48 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:20:48.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:48 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:48.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:48 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:48.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:48 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:48.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:48 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T14:20:48.843 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:48 smithi194 ceph-mon[45450]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:48.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:48 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:48.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:48 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:48.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:48 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:20:48.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:48 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:48.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:48 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:48.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:48 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:48.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:48 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T14:20:48.861 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.859+0000 7f6300c1a700 1 -- 172.21.15.194:0/259542543 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f62fc1971f0 con 0x7f62fc19ad80 2024-01-24T14:20:48.862 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.859+0000 7f62eb7fe700 1 -- 172.21.15.194:0/259542543 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1274 (secure 0 0 0) 0x7f62f401b8d0 con 0x7f62fc19ad80 2024-01-24T14:20:48.862 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-24T14:20:48.862 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":3,"fsid":"654b0c8e-bac3-11ee-95b1-87774f69a715","modified":"2024-01-24T14:20:42.317232Z","created":"2024-01-24T14:19:08.559591Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:3300","nonce":0},{"type":"v1","addr":"172.21.15.19:6789","nonce":0}]},"addr":"172.21.15.19:6789/0","public_addr":"172.21.15.19:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-01-24T14:20:48.863 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.861+0000 7f6300c1a700 1 -- 172.21.15.194:0/259542543 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f62dc038870 msgr2=0x7f62dc03ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:48.863 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.861+0000 7f6300c1a700 1 --2- 172.21.15.194:0/259542543 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f62dc038870 0x7f62dc03ad30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f62ec005ed0 tx=0x7f62ec005e60 comp rx=0 tx=0).stop 2024-01-24T14:20:48.863 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.861+0000 7f6300c1a700 1 -- 172.21.15.194:0/259542543 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62fc19ad80 msgr2=0x7f62fc19b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:48.863 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.861+0000 7f6300c1a700 1 --2- 172.21.15.194:0/259542543 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62fc19ad80 0x7f62fc19b230 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f62f400bd50 tx=0x7f62f400be30 comp rx=0 tx=0).stop 2024-01-24T14:20:48.863 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.861+0000 7f6300c1a700 1 -- 172.21.15.194:0/259542543 shutdown_connections 2024-01-24T14:20:48.863 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.861+0000 7f6300c1a700 1 --2- 172.21.15.194:0/259542543 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f62dc038870 0x7f62dc03ad30 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:48.863 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.861+0000 7f6300c1a700 1 --2- 172.21.15.194:0/259542543 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62fc19ad80 0x7f62fc19b230 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:48.863 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.861+0000 7f6300c1a700 1 --2- 172.21.15.194:0/259542543 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f62e4006370 0x7f62fc195e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:48.863 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.861+0000 7f6300c1a700 1 --2- 172.21.15.194:0/259542543 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f62fc1963d0 0x7f62fc19a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:48.863 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.862+0000 7f6300c1a700 1 -- 172.21.15.194:0/259542543 >> 172.21.15.194:0/259542543 conn(0x7f62fc0fb300 msgr2=0x7f62fc1029a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:20:48.864 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.862+0000 7f6300c1a700 1 -- 172.21.15.194:0/259542543 shutdown_connections 2024-01-24T14:20:48.864 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:48.862+0000 7f6300c1a700 1 -- 172.21.15.194:0/259542543 wait complete. 2024-01-24T14:20:48.864 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 3 2024-01-24T14:20:49.222 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-01-24T14:20:49.223 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 -- ceph config generate-minimal-conf 2024-01-24T14:20:49.491 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/mon.a/config 2024-01-24T14:20:49.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:49 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-01-24T14:20:49.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:49 smithi123 ceph-mon[44966]: mon.a calling monitor election 2024-01-24T14:20:49.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:49 smithi123 ceph-mon[44966]: mon.c calling monitor election 2024-01-24T14:20:49.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:49 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T14:20:49.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:49 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T14:20:49.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:49 smithi123 ceph-mon[44966]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:49.801 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:49 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T14:20:49.801 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:49 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T14:20:49.801 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:49 smithi123 ceph-mon[44966]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:49.801 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:49 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T14:20:49.801 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:49 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T14:20:49.801 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:49 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T14:20:49.801 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:49 smithi123 ceph-mon[44966]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-01-24T14:20:49.801 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:49 smithi123 ceph-mon[44966]: monmap e3: 3 mons at {a=[v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0],b=[v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0],c=[v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-01-24T14:20:49.801 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:49 smithi123 ceph-mon[44966]: fsmap 2024-01-24T14:20:49.801 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:49 smithi123 ceph-mon[44966]: osdmap e4: 0 total, 0 up, 0 in 2024-01-24T14:20:49.801 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:49 smithi123 ceph-mon[44966]: mgrmap e13: x(active, since 44s) 2024-01-24T14:20:49.802 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:49 smithi123 ceph-mon[44966]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-01-24T14:20:49.802 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:49 smithi123 ceph-mon[44966]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-01-24T14:20:49.802 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:49 smithi123 ceph-mon[44966]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-01-24T14:20:49.802 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:49 smithi123 ceph-mon[44966]: mon.b (rank 2) addr [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] is down (out of quorum) 2024-01-24T14:20:49.802 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:49 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:49.802 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:49 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:49.802 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:49 smithi123 ceph-mon[44966]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:49.802 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:49 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:49.802 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:49 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:49.802 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:49 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:20:49.802 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:49 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:49.803 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:49 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:49.803 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:49 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:49.803 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:49 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T14:20:50.485 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:50 smithi019 ceph-mon[40663]: mon.b calling monitor election 2024-01-24T14:20:50.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:50 smithi019 ceph-mon[40663]: Updating smithi019:/etc/ceph/ceph.conf 2024-01-24T14:20:50.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:50 smithi019 ceph-mon[40663]: Updating smithi123:/etc/ceph/ceph.conf 2024-01-24T14:20:50.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:50 smithi019 ceph-mon[40663]: Updating smithi194:/etc/ceph/ceph.conf 2024-01-24T14:20:50.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:50 smithi019 ceph-mon[40663]: Updating smithi123:/var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/config/ceph.conf 2024-01-24T14:20:50.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:50 smithi019 ceph-mon[40663]: Updating smithi194:/var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/config/ceph.conf 2024-01-24T14:20:50.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:50 smithi019 ceph-mon[40663]: Updating smithi019:/var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/config/ceph.conf 2024-01-24T14:20:50.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:50 smithi019 ceph-mon[40663]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:50.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:50 smithi019 ceph-mon[40663]: Reconfiguring mon.a (unknown last config time)... 2024-01-24T14:20:50.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:50 smithi019 ceph-mon[40663]: Reconfiguring daemon mon.a on smithi019 2024-01-24T14:20:50.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:50 smithi019 ceph-mon[40663]: mon.b calling monitor election 2024-01-24T14:20:50.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:50 smithi019 ceph-mon[40663]: mon.c calling monitor election 2024-01-24T14:20:50.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:50 smithi019 ceph-mon[40663]: mon.a calling monitor election 2024-01-24T14:20:50.487 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:50 smithi019 ceph-mon[40663]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-01-24T14:20:50.487 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:50 smithi019 ceph-mon[40663]: monmap e3: 3 mons at {a=[v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0],b=[v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0],c=[v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-01-24T14:20:50.487 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:50 smithi019 ceph-mon[40663]: fsmap 2024-01-24T14:20:50.487 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:50 smithi019 ceph-mon[40663]: osdmap e4: 0 total, 0 up, 0 in 2024-01-24T14:20:50.487 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:50 smithi019 ceph-mon[40663]: mgrmap e13: x(active, since 46s) 2024-01-24T14:20:50.487 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:50 smithi019 ceph-mon[40663]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-01-24T14:20:50.487 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:50 smithi019 ceph-mon[40663]: Cluster is now healthy 2024-01-24T14:20:50.487 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:50 smithi019 ceph-mon[40663]: overall HEALTH_OK 2024-01-24T14:20:50.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:50 smithi123 ceph-mon[44966]: mon.b calling monitor election 2024-01-24T14:20:50.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:50 smithi123 ceph-mon[44966]: Updating smithi019:/etc/ceph/ceph.conf 2024-01-24T14:20:50.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:50 smithi123 ceph-mon[44966]: Updating smithi123:/etc/ceph/ceph.conf 2024-01-24T14:20:50.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:50 smithi123 ceph-mon[44966]: Updating smithi194:/etc/ceph/ceph.conf 2024-01-24T14:20:50.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:50 smithi123 ceph-mon[44966]: Updating smithi123:/var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/config/ceph.conf 2024-01-24T14:20:50.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:50 smithi123 ceph-mon[44966]: Updating smithi194:/var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/config/ceph.conf 2024-01-24T14:20:50.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:50 smithi123 ceph-mon[44966]: Updating smithi019:/var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/config/ceph.conf 2024-01-24T14:20:50.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:50 smithi123 ceph-mon[44966]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:50.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:50 smithi123 ceph-mon[44966]: Reconfiguring mon.a (unknown last config time)... 2024-01-24T14:20:50.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:50 smithi123 ceph-mon[44966]: Reconfiguring daemon mon.a on smithi019 2024-01-24T14:20:50.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:50 smithi123 ceph-mon[44966]: mon.b calling monitor election 2024-01-24T14:20:50.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:50 smithi123 ceph-mon[44966]: mon.c calling monitor election 2024-01-24T14:20:50.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:50 smithi123 ceph-mon[44966]: mon.a calling monitor election 2024-01-24T14:20:50.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:50 smithi123 ceph-mon[44966]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-01-24T14:20:50.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:50 smithi123 ceph-mon[44966]: monmap e3: 3 mons at {a=[v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0],b=[v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0],c=[v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-01-24T14:20:50.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:50 smithi123 ceph-mon[44966]: fsmap 2024-01-24T14:20:50.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:50 smithi123 ceph-mon[44966]: osdmap e4: 0 total, 0 up, 0 in 2024-01-24T14:20:50.551 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:50 smithi123 ceph-mon[44966]: mgrmap e13: x(active, since 46s) 2024-01-24T14:20:50.551 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:50 smithi123 ceph-mon[44966]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-01-24T14:20:50.551 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:50 smithi123 ceph-mon[44966]: Cluster is now healthy 2024-01-24T14:20:50.551 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:50 smithi123 ceph-mon[44966]: overall HEALTH_OK 2024-01-24T14:20:50.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:50 smithi194 ceph-mon[45450]: mon.b calling monitor election 2024-01-24T14:20:50.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:50 smithi194 ceph-mon[45450]: Updating smithi019:/etc/ceph/ceph.conf 2024-01-24T14:20:50.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:50 smithi194 ceph-mon[45450]: Updating smithi123:/etc/ceph/ceph.conf 2024-01-24T14:20:50.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:50 smithi194 ceph-mon[45450]: Updating smithi194:/etc/ceph/ceph.conf 2024-01-24T14:20:50.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:50 smithi194 ceph-mon[45450]: Updating smithi123:/var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/config/ceph.conf 2024-01-24T14:20:50.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:50 smithi194 ceph-mon[45450]: Updating smithi194:/var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/config/ceph.conf 2024-01-24T14:20:50.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:50 smithi194 ceph-mon[45450]: Updating smithi019:/var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/config/ceph.conf 2024-01-24T14:20:50.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:50 smithi194 ceph-mon[45450]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:50.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:50 smithi194 ceph-mon[45450]: Reconfiguring mon.a (unknown last config time)... 2024-01-24T14:20:50.595 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:50 smithi194 ceph-mon[45450]: Reconfiguring daemon mon.a on smithi019 2024-01-24T14:20:50.595 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:50 smithi194 ceph-mon[45450]: mon.b calling monitor election 2024-01-24T14:20:50.595 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:50 smithi194 ceph-mon[45450]: mon.c calling monitor election 2024-01-24T14:20:50.595 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:50 smithi194 ceph-mon[45450]: mon.a calling monitor election 2024-01-24T14:20:50.595 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:50 smithi194 ceph-mon[45450]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-01-24T14:20:50.595 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:50 smithi194 ceph-mon[45450]: monmap e3: 3 mons at {a=[v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0],b=[v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0],c=[v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-01-24T14:20:50.595 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:50 smithi194 ceph-mon[45450]: fsmap 2024-01-24T14:20:50.595 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:50 smithi194 ceph-mon[45450]: osdmap e4: 0 total, 0 up, 0 in 2024-01-24T14:20:50.595 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:50 smithi194 ceph-mon[45450]: mgrmap e13: x(active, since 46s) 2024-01-24T14:20:50.595 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:50 smithi194 ceph-mon[45450]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-01-24T14:20:50.595 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:50 smithi194 ceph-mon[45450]: Cluster is now healthy 2024-01-24T14:20:50.595 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:50 smithi194 ceph-mon[45450]: overall HEALTH_OK 2024-01-24T14:20:50.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:50.789+0000 7f8b35f26700 1 -- 172.21.15.19:0/246679092 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8b30068860 msgr2=0x7f8b30068ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:50.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:50.789+0000 7f8b35f26700 1 --2- 172.21.15.19:0/246679092 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8b30068860 0x7f8b30068ce0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8b18009990 tx=0x7f8b18009ca0 comp rx=0 tx=0).stop 2024-01-24T14:20:50.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:50.789+0000 7f8b35f26700 1 -- 172.21.15.19:0/246679092 shutdown_connections 2024-01-24T14:20:50.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:50.789+0000 7f8b35f26700 1 --2- 172.21.15.19:0/246679092 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8b30069220 0x7f8b3010db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:50.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:50.789+0000 7f8b35f26700 1 --2- 172.21.15.19:0/246679092 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b30106ed0 0x7f8b301072b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:50.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:50.789+0000 7f8b35f26700 1 --2- 172.21.15.19:0/246679092 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8b30068860 0x7f8b30068ce0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:50.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:50.789+0000 7f8b35f26700 1 -- 172.21.15.19:0/246679092 >> 172.21.15.19:0/246679092 conn(0x7f8b30076d40 msgr2=0x7f8b30077150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:20:50.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:50.789+0000 7f8b35f26700 1 -- 172.21.15.19:0/246679092 shutdown_connections 2024-01-24T14:20:50.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:50.789+0000 7f8b35f26700 1 -- 172.21.15.19:0/246679092 wait complete. 2024-01-24T14:20:50.792 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:50.790+0000 7f8b35f26700 1 Processor -- start 2024-01-24T14:20:50.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:50.790+0000 7f8b35f26700 1 -- start start 2024-01-24T14:20:50.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:50.790+0000 7f8b35f26700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8b30068860 0x7f8b30196e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:50.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:50.790+0000 7f8b35f26700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8b30069220 0x7f8b30197350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:50.795 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:50.791+0000 7f8b35f26700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b30106ed0 0x7f8b3019b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:50.795 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:50.791+0000 7f8b35f26700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3019bdd0 con 0x7f8b30069220 2024-01-24T14:20:50.795 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:50.791+0000 7f8b35f26700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3019bf40 con 0x7f8b30106ed0 2024-01-24T14:20:50.795 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:50.791+0000 7f8b35f26700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3019c100 con 0x7f8b30068860 2024-01-24T14:20:50.795 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:50.791+0000 7f8b2effd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8b30069220 0x7f8b30197350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:50.795 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:50.791+0000 7f8b2effd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8b30069220 0x7f8b30197350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:42298/0 (socket says 172.21.15.19:42298) 2024-01-24T14:20:50.795 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:50.791+0000 7f8b2effd700 1 -- 172.21.15.19:0/24506296 learned_addr learned my addr 172.21.15.19:0/24506296 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:20:50.795 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:50.791+0000 7f8b2ffff700 1 --2- 172.21.15.19:0/24506296 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b30106ed0 0x7f8b3019b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:50.795 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:50.791+0000 7f8b2f7fe700 1 --2- 172.21.15.19:0/24506296 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8b30068860 0x7f8b30196e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:50.795 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:50.791+0000 7f8b2effd700 1 -- 172.21.15.19:0/24506296 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8b30068860 msgr2=0x7f8b30196e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:50.795 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:50.791+0000 7f8b2effd700 1 --2- 172.21.15.19:0/24506296 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8b30068860 0x7f8b30196e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:50.795 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:50.791+0000 7f8b2effd700 1 -- 172.21.15.19:0/24506296 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b30106ed0 msgr2=0x7f8b3019b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:50.795 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:50.791+0000 7f8b2effd700 1 --2- 172.21.15.19:0/24506296 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b30106ed0 0x7f8b3019b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:50.795 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:50.791+0000 7f8b2effd700 1 -- 172.21.15.19:0/24506296 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b180095f0 con 0x7f8b30069220 2024-01-24T14:20:50.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:50.791+0000 7f8b2f7fe700 1 --2- 172.21.15.19:0/24506296 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8b30068860 0x7f8b30196e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T14:20:50.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:50.791+0000 7f8b2effd700 1 --2- 172.21.15.19:0/24506296 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8b30069220 0x7f8b30197350 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8b180095c0 tx=0x7f8b1800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:20:50.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:50.791+0000 7f8b2cff9700 1 -- 172.21.15.19:0/24506296 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b1801d070 con 0x7f8b30069220 2024-01-24T14:20:50.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:50.791+0000 7f8b35f26700 1 -- 172.21.15.19:0/24506296 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b3019c380 con 0x7f8b30069220 2024-01-24T14:20:50.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:50.791+0000 7f8b2cff9700 1 -- 172.21.15.19:0/24506296 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8b180054f0 con 0x7f8b30069220 2024-01-24T14:20:50.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:50.791+0000 7f8b2cff9700 1 -- 172.21.15.19:0/24506296 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b18004b30 con 0x7f8b30069220 2024-01-24T14:20:50.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:50.791+0000 7f8b35f26700 1 -- 172.21.15.19:0/24506296 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b301a3c40 con 0x7f8b30069220 2024-01-24T14:20:50.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:50.792+0000 7f8b2cff9700 1 -- 172.21.15.19:0/24506296 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 13) v1 ==== 45324+0+0 (secure 0 0 0) 0x7f8b18004c90 con 0x7f8b30069220 2024-01-24T14:20:50.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:50.793+0000 7f8b2cff9700 1 --2- 172.21.15.19:0/24506296 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f8b1c0385d0 0x7f8b1c03aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:50.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:50.793+0000 7f8b2cff9700 1 -- 172.21.15.19:0/24506296 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f8b1804bcf0 con 0x7f8b30069220 2024-01-24T14:20:50.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:50.793+0000 7f8b35f26700 1 -- 172.21.15.19:0/24506296 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b3004ea40 con 0x7f8b30069220 2024-01-24T14:20:50.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:50.793+0000 7f8b2f7fe700 1 --2- 172.21.15.19:0/24506296 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f8b1c0385d0 0x7f8b1c03aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:50.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:50.793+0000 7f8b2f7fe700 1 --2- 172.21.15.19:0/24506296 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f8b1c0385d0 0x7f8b1c03aa90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8b20005fd0 tx=0x7f8b20005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:20:50.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:50.798+0000 7f8b2cff9700 1 -- 172.21.15.19:0/24506296 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f8b18026070 con 0x7f8b30069220 2024-01-24T14:20:51.007 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:51.006+0000 7f8b35f26700 1 -- 172.21.15.19:0/24506296 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f8b301a3dd0 con 0x7f8b30069220 2024-01-24T14:20:51.007 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:51.006+0000 7f8b2cff9700 1 -- 172.21.15.19:0/24506296 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+275 (secure 0 0 0) 0x7f8b1802bce0 con 0x7f8b30069220 2024-01-24T14:20:51.008 INFO:teuthology.orchestra.run.smithi019.stdout:# minimal ceph.conf for 654b0c8e-bac3-11ee-95b1-87774f69a715 2024-01-24T14:20:51.008 INFO:teuthology.orchestra.run.smithi019.stdout:[global] 2024-01-24T14:20:51.008 INFO:teuthology.orchestra.run.smithi019.stdout: fsid = 654b0c8e-bac3-11ee-95b1-87774f69a715 2024-01-24T14:20:51.008 INFO:teuthology.orchestra.run.smithi019.stdout: mon_host = [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-01-24T14:20:51.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:51.009+0000 7f8b35f26700 1 -- 172.21.15.19:0/24506296 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f8b1c0385d0 msgr2=0x7f8b1c03aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:51.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:51.009+0000 7f8b35f26700 1 --2- 172.21.15.19:0/24506296 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f8b1c0385d0 0x7f8b1c03aa90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8b20005fd0 tx=0x7f8b20005ee0 comp rx=0 tx=0).stop 2024-01-24T14:20:51.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:51.009+0000 7f8b35f26700 1 -- 172.21.15.19:0/24506296 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8b30069220 msgr2=0x7f8b30197350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:51.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:51.009+0000 7f8b35f26700 1 --2- 172.21.15.19:0/24506296 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8b30069220 0x7f8b30197350 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8b180095c0 tx=0x7f8b1800f690 comp rx=0 tx=0).stop 2024-01-24T14:20:51.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:51.009+0000 7f8b35f26700 1 -- 172.21.15.19:0/24506296 shutdown_connections 2024-01-24T14:20:51.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:51.009+0000 7f8b35f26700 1 --2- 172.21.15.19:0/24506296 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b30106ed0 0x7f8b3019b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:51.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:51.009+0000 7f8b35f26700 1 --2- 172.21.15.19:0/24506296 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8b30068860 0x7f8b30196e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:51.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:51.009+0000 7f8b35f26700 1 --2- 172.21.15.19:0/24506296 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f8b1c0385d0 0x7f8b1c03aa90 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:51.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:51.009+0000 7f8b35f26700 1 --2- 172.21.15.19:0/24506296 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8b30069220 0x7f8b30197350 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:51.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:51.009+0000 7f8b35f26700 1 -- 172.21.15.19:0/24506296 >> 172.21.15.19:0/24506296 conn(0x7f8b30076d40 msgr2=0x7f8b300fe7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:20:51.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:51.009+0000 7f8b35f26700 1 -- 172.21.15.19:0/24506296 shutdown_connections 2024-01-24T14:20:51.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:51.009+0000 7f8b35f26700 1 -- 172.21.15.19:0/24506296 wait complete. 2024-01-24T14:20:51.305 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:51 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T14:20:51.306 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:51 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:51.306 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:51 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:51.306 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:51 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-24T14:20:51.306 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:51 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-24T14:20:51.306 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:51 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:20:51.306 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:51 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/24506296' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:20:51.389 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-01-24T14:20:51.390 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-01-24T14:20:51.390 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/ceph/ceph.conf 2024-01-24T14:20:51.427 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-01-24T14:20:51.427 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-24T14:20:51.482 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:51 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T14:20:51.482 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:51 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:51.482 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:51 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:51.482 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:51 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-24T14:20:51.482 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:51 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-24T14:20:51.483 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:51 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:20:51.483 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:51 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/24506296' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:20:51.504 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-01-24T14:20:51.505 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/ceph/ceph.conf 2024-01-24T14:20:51.536 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-01-24T14:20:51.536 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-24T14:20:51.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:51 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T14:20:51.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:51 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:51.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:51 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:51.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:51 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-24T14:20:51.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:51 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-24T14:20:51.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:51 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:20:51.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:51 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/24506296' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:20:51.603 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-01-24T14:20:51.603 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/ceph/ceph.conf 2024-01-24T14:20:51.640 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-01-24T14:20:51.640 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-24T14:20:51.711 INFO:tasks.cephadm:Adding mgr.x on smithi019 2024-01-24T14:20:51.711 INFO:tasks.cephadm:Adding mgr.y on smithi123 2024-01-24T14:20:51.711 INFO:tasks.cephadm:Adding mgr.z on smithi194 2024-01-24T14:20:51.712 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 -- ceph orch apply mgr '3;smithi019=x;smithi123=y;smithi194=z' 2024-01-24T14:20:51.973 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/mon.c/config 2024-01-24T14:20:52.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:52 smithi194 ceph-mon[45450]: Reconfiguring mon.b (monmap changed)... 2024-01-24T14:20:52.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:52 smithi194 ceph-mon[45450]: Reconfiguring daemon mon.b on smithi123 2024-01-24T14:20:52.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:52 smithi194 ceph-mon[45450]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:52.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:52 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:52.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:52 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:52.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:52 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-24T14:20:52.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:52 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-24T14:20:52.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:52 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:20:52.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:52 smithi123 ceph-mon[44966]: Reconfiguring mon.b (monmap changed)... 2024-01-24T14:20:52.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:52 smithi123 ceph-mon[44966]: Reconfiguring daemon mon.b on smithi123 2024-01-24T14:20:52.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:52 smithi123 ceph-mon[44966]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:52.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:52 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:52.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:52 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:52.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:52 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-24T14:20:52.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:52 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-24T14:20:52.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:52 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:20:52.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:52 smithi019 ceph-mon[40663]: Reconfiguring mon.b (monmap changed)... 2024-01-24T14:20:52.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:52 smithi019 ceph-mon[40663]: Reconfiguring daemon mon.b on smithi123 2024-01-24T14:20:52.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:52 smithi019 ceph-mon[40663]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:52.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:52 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:52.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:52 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:52.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:52 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-01-24T14:20:52.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:52 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-01-24T14:20:52.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:52 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:20:53.243 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.241+0000 7fec91712700 1 -- 172.21.15.194:0/1167019042 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fec8c0ff650 msgr2=0x7fec8c0ffa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:53.243 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.241+0000 7fec91712700 1 --2- 172.21.15.194:0/1167019042 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fec8c0ff650 0x7fec8c0ffa70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fec7c009990 tx=0x7fec7c009ca0 comp rx=0 tx=0).stop 2024-01-24T14:20:53.243 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.241+0000 7fec91712700 1 -- 172.21.15.194:0/1167019042 shutdown_connections 2024-01-24T14:20:53.243 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.241+0000 7fec91712700 1 --2- 172.21.15.194:0/1167019042 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec8c100970 0x7fec8c100df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:53.244 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.241+0000 7fec91712700 1 --2- 172.21.15.194:0/1167019042 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fec8c0ff650 0x7fec8c0ffa70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:53.244 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.241+0000 7fec91712700 1 --2- 172.21.15.194:0/1167019042 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fec8c0fffb0 0x7fec8c100430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:53.244 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.241+0000 7fec91712700 1 -- 172.21.15.194:0/1167019042 >> 172.21.15.194:0/1167019042 conn(0x7fec8c0fb250 msgr2=0x7fec8c0fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:20:53.244 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.241+0000 7fec91712700 1 -- 172.21.15.194:0/1167019042 shutdown_connections 2024-01-24T14:20:53.244 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.241+0000 7fec91712700 1 -- 172.21.15.194:0/1167019042 wait complete. 2024-01-24T14:20:53.244 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.241+0000 7fec91712700 1 Processor -- start 2024-01-24T14:20:53.244 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.241+0000 7fec91712700 1 -- start start 2024-01-24T14:20:53.244 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.242+0000 7fec91712700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fec8c0ff650 0x7fec8c198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:53.244 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.242+0000 7fec91712700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec8c0fffb0 0x7fec8c1991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:53.244 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.242+0000 7fec91712700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fec8c100970 0x7fec8c1a0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:53.244 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.242+0000 7fec91712700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8c1a0810 con 0x7fec8c100970 2024-01-24T14:20:53.245 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.242+0000 7fec91712700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8c1a0950 con 0x7fec8c0fffb0 2024-01-24T14:20:53.245 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.242+0000 7fec91712700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8c1a0b10 con 0x7fec8c0ff650 2024-01-24T14:20:53.245 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.242+0000 7fec8affd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fec8c0ff650 0x7fec8c198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:53.245 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.242+0000 7fec8affd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fec8c0ff650 0x7fec8c198c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:42648/0 (socket says 172.21.15.194:42648) 2024-01-24T14:20:53.245 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.242+0000 7fec8affd700 1 -- 172.21.15.194:0/1416640211 learned_addr learned my addr 172.21.15.194:0/1416640211 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-24T14:20:53.245 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.242+0000 7fec8a7fc700 1 --2- 172.21.15.194:0/1416640211 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec8c0fffb0 0x7fec8c1991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:53.245 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.242+0000 7fec8b7fe700 1 --2- 172.21.15.194:0/1416640211 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fec8c100970 0x7fec8c1a0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:53.245 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.242+0000 7fec8affd700 1 -- 172.21.15.194:0/1416640211 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec8c0fffb0 msgr2=0x7fec8c1991d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:53.245 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.242+0000 7fec8affd700 1 --2- 172.21.15.194:0/1416640211 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec8c0fffb0 0x7fec8c1991d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:53.245 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.242+0000 7fec8affd700 1 -- 172.21.15.194:0/1416640211 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fec8c100970 msgr2=0x7fec8c1a0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:53.245 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.242+0000 7fec8affd700 1 --2- 172.21.15.194:0/1416640211 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fec8c100970 0x7fec8c1a0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:53.245 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.242+0000 7fec8affd700 1 -- 172.21.15.194:0/1416640211 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec7c0095f0 con 0x7fec8c0ff650 2024-01-24T14:20:53.246 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.242+0000 7fec8b7fe700 1 --2- 172.21.15.194:0/1416640211 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fec8c100970 0x7fec8c1a0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T14:20:53.246 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.242+0000 7fec8affd700 1 --2- 172.21.15.194:0/1416640211 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fec8c0ff650 0x7fec8c198c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fec7c005910 tx=0x7fec7c00f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:20:53.247 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.244+0000 7fec73fff700 1 -- 172.21.15.194:0/1416640211 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7c01d070 con 0x7fec8c0ff650 2024-01-24T14:20:53.247 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.244+0000 7fec73fff700 1 -- 172.21.15.194:0/1416640211 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fec7c00fb80 con 0x7fec8c0ff650 2024-01-24T14:20:53.247 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.244+0000 7fec91712700 1 -- 172.21.15.194:0/1416640211 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec8c1a0d90 con 0x7fec8c0ff650 2024-01-24T14:20:53.247 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.245+0000 7fec73fff700 1 -- 172.21.15.194:0/1416640211 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7c017650 con 0x7fec8c0ff650 2024-01-24T14:20:53.247 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.245+0000 7fec91712700 1 -- 172.21.15.194:0/1416640211 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec8c1a11b0 con 0x7fec8c0ff650 2024-01-24T14:20:53.248 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.245+0000 7fec73fff700 1 -- 172.21.15.194:0/1416640211 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45324+0+0 (secure 0 0 0) 0x7fec7c0177b0 con 0x7fec8c0ff650 2024-01-24T14:20:53.248 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.245+0000 7fec73fff700 1 --2- 172.21.15.194:0/1416640211 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fec740385d0 0x7fec7403aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:53.248 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.245+0000 7fec73fff700 1 -- 172.21.15.194:0/1416640211 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fec7c04c5a0 con 0x7fec8c0ff650 2024-01-24T14:20:53.248 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.245+0000 7fec91712700 1 -- 172.21.15.194:0/1416640211 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec8c10a430 con 0x7fec8c0ff650 2024-01-24T14:20:53.248 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.246+0000 7fec8a7fc700 1 --2- 172.21.15.194:0/1416640211 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fec740385d0 0x7fec7403aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:53.248 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.246+0000 7fec8a7fc700 1 --2- 172.21.15.194:0/1416640211 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fec740385d0 0x7fec7403aa90 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fec8c19a1a0 tx=0x7fec8000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:20:53.250 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.250+0000 7fec73fff700 1 -- 172.21.15.194:0/1416640211 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fec7c027070 con 0x7fec8c0ff650 2024-01-24T14:20:53.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:53 smithi194 ceph-mon[45450]: Reconfiguring mon.c (monmap changed)... 2024-01-24T14:20:53.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:53 smithi194 ceph-mon[45450]: Reconfiguring daemon mon.c on smithi194 2024-01-24T14:20:53.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:53 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:53.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:53 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:53.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:53 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:20:53.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:53 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:20:53.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:53 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:20:53.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:53 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:53.450 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.450+0000 7fec91712700 1 -- 172.21.15.194:0/1416640211 --> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi019=x;smithi123=y;smithi194=z", "target": ["mon-mgr", ""]}) v1 -- 0x7fec8c061210 con 0x7fec740385d0 2024-01-24T14:20:53.457 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.457+0000 7fec73fff700 1 -- 172.21.15.194:0/1416640211 <== mgr.14152 v2:172.21.15.19:6800/1634350318 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fec8c061210 con 0x7fec740385d0 2024-01-24T14:20:53.458 INFO:teuthology.orchestra.run.smithi194.stdout:Scheduled mgr update... 2024-01-24T14:20:53.459 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.459+0000 7fec91712700 1 -- 172.21.15.194:0/1416640211 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fec740385d0 msgr2=0x7fec7403aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:53.460 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.459+0000 7fec91712700 1 --2- 172.21.15.194:0/1416640211 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fec740385d0 0x7fec7403aa90 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fec8c19a1a0 tx=0x7fec8000b410 comp rx=0 tx=0).stop 2024-01-24T14:20:53.460 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.459+0000 7fec91712700 1 -- 172.21.15.194:0/1416640211 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fec8c0ff650 msgr2=0x7fec8c198c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:53.460 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.459+0000 7fec91712700 1 --2- 172.21.15.194:0/1416640211 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fec8c0ff650 0x7fec8c198c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fec7c005910 tx=0x7fec7c00f690 comp rx=0 tx=0).stop 2024-01-24T14:20:53.460 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.459+0000 7fec91712700 1 -- 172.21.15.194:0/1416640211 shutdown_connections 2024-01-24T14:20:53.460 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.459+0000 7fec91712700 1 --2- 172.21.15.194:0/1416640211 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fec740385d0 0x7fec7403aa90 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:53.460 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.459+0000 7fec91712700 1 --2- 172.21.15.194:0/1416640211 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fec8c100970 0x7fec8c1a0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:53.460 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.459+0000 7fec91712700 1 --2- 172.21.15.194:0/1416640211 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fec8c0ff650 0x7fec8c198c90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:53.460 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.459+0000 7fec91712700 1 --2- 172.21.15.194:0/1416640211 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec8c0fffb0 0x7fec8c1991d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:53.460 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.459+0000 7fec91712700 1 -- 172.21.15.194:0/1416640211 >> 172.21.15.194:0/1416640211 conn(0x7fec8c0fb250 msgr2=0x7fec8c0fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:20:53.460 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.459+0000 7fec91712700 1 -- 172.21.15.194:0/1416640211 shutdown_connections 2024-01-24T14:20:53.460 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:20:53.459+0000 7fec91712700 1 -- 172.21.15.194:0/1416640211 wait complete. 2024-01-24T14:20:53.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:53 smithi123 ceph-mon[44966]: Reconfiguring mon.c (monmap changed)... 2024-01-24T14:20:53.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:53 smithi123 ceph-mon[44966]: Reconfiguring daemon mon.c on smithi194 2024-01-24T14:20:53.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:53 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:53.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:53 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:53.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:53 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:20:53.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:53 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:20:53.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:53 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:20:53.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:53 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:53.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:53 smithi019 ceph-mon[40663]: Reconfiguring mon.c (monmap changed)... 2024-01-24T14:20:53.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:53 smithi019 ceph-mon[40663]: Reconfiguring daemon mon.c on smithi194 2024-01-24T14:20:53.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:53 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:53.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:53 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:53.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:53 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:20:53.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:53 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:20:53.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:53 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:20:53.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:53 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:53.907 DEBUG:teuthology.orchestra.run.smithi123:mgr.y> sudo journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mgr.y.service 2024-01-24T14:20:53.909 DEBUG:teuthology.orchestra.run.smithi194:mgr.z> sudo journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mgr.z.service 2024-01-24T14:20:53.911 INFO:tasks.cephadm:Deploying OSDs... 2024-01-24T14:20:53.911 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-01-24T14:20:53.912 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/scratch_devs of=/dev/stdout 2024-01-24T14:20:53.934 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-24T14:20:53.934 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_1 2024-01-24T14:20:53.940 INFO:journalctl@ceph.mgr.z.smithi194.stdout:-- Logs begin at Wed 2024-01-24 14:04:01 UTC. -- 2024-01-24T14:20:53.943 INFO:journalctl@ceph.mgr.y.smithi123.stdout:-- Logs begin at Wed 2024-01-24 14:03:45 UTC. -- 2024-01-24T14:20:53.997 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-24T14:20:53.997 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-24T14:20:53.997 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 6h/6d Inode: 172973 Links: 1 2024-01-24T14:20:53.997 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-24T14:20:53.997 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2024-01-24T14:20:53.998 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2024-01-24 14:20:12.553097864 +0000 2024-01-24T14:20:53.998 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2024-01-24 14:13:56.873919985 +0000 2024-01-24T14:20:53.998 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2024-01-24 14:13:56.873919985 +0000 2024-01-24T14:20:53.998 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: - 2024-01-24T14:20:53.998 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-24T14:20:54.073 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2024-01-24T14:20:54.073 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2024-01-24T14:20:54.073 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000276355 s, 1.9 MB/s 2024-01-24T14:20:54.074 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-24T14:20:54.139 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_2 2024-01-24T14:20:54.201 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-24T14:20:54.202 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-24T14:20:54.202 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 6h/6d Inode: 173593 Links: 1 2024-01-24T14:20:54.202 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-24T14:20:54.202 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2024-01-24T14:20:54.202 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2024-01-24 14:20:12.614097244 +0000 2024-01-24T14:20:54.202 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2024-01-24 14:13:57.259916096 +0000 2024-01-24T14:20:54.202 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2024-01-24 14:13:57.259916096 +0000 2024-01-24T14:20:54.202 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: - 2024-01-24T14:20:54.202 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-24T14:20:54.275 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2024-01-24T14:20:54.275 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2024-01-24T14:20:54.275 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.00024103 s, 2.1 MB/s 2024-01-24T14:20:54.276 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-24T14:20:54.340 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_3 2024-01-24T14:20:54.403 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-24T14:20:54.403 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-24T14:20:54.403 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 6h/6d Inode: 177616 Links: 1 2024-01-24T14:20:54.403 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-24T14:20:54.403 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2024-01-24T14:20:54.403 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2024-01-24 14:20:12.674096634 +0000 2024-01-24T14:20:54.403 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2024-01-24 14:13:57.638912278 +0000 2024-01-24T14:20:54.403 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2024-01-24 14:13:57.638912278 +0000 2024-01-24T14:20:54.404 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: - 2024-01-24T14:20:54.404 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-24T14:20:54.474 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:54 smithi019 ceph-mon[40663]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:54.474 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:54 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:54.474 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:54 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:20:54.474 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:54 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:20:54.474 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:54 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:20:54.474 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:54 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:54.474 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:54 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-24T14:20:54.475 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:54 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-01-24T14:20:54.475 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:54 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-24T14:20:54.475 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:54 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:20:54.477 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2024-01-24T14:20:54.477 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2024-01-24T14:20:54.477 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000217685 s, 2.4 MB/s 2024-01-24T14:20:54.478 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-24T14:20:54.542 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_4 2024-01-24T14:20:54.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:54 smithi123 ceph-mon[44966]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:54.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:54 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:54.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:54 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:20:54.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:54 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:20:54.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:54 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:20:54.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:54 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:54.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:54 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-24T14:20:54.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:54 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-01-24T14:20:54.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:54 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-24T14:20:54.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:54 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:20:54.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:54 smithi194 ceph-mon[45450]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:54.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:54 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:54.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:54 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:20:54.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:54 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:20:54.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:54 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:20:54.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:54 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:54.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:54 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-24T14:20:54.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:54 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-01-24T14:20:54.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:54 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-24T14:20:54.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:54 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:20:54.606 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-24T14:20:54.606 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-24T14:20:54.606 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 6h/6d Inode: 177679 Links: 1 2024-01-24T14:20:54.606 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-24T14:20:54.606 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2024-01-24T14:20:54.606 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2024-01-24 14:20:12.733096034 +0000 2024-01-24T14:20:54.606 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2024-01-24 14:13:58.012908511 +0000 2024-01-24T14:20:54.606 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2024-01-24 14:13:58.012908511 +0000 2024-01-24T14:20:54.606 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: - 2024-01-24T14:20:54.607 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-24T14:20:54.679 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2024-01-24T14:20:54.679 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2024-01-24T14:20:54.679 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000231526 s, 2.2 MB/s 2024-01-24T14:20:54.680 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-24T14:20:54.744 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-01-24T14:20:54.744 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/scratch_devs of=/dev/stdout 2024-01-24T14:20:54.766 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-24T14:20:54.766 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_1 2024-01-24T14:20:54.827 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-24T14:20:54.827 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-24T14:20:54.827 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 6h/6d Inode: 175565 Links: 1 2024-01-24T14:20:54.828 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-24T14:20:54.828 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2024-01-24T14:20:54.828 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2024-01-24 14:20:41.879880142 +0000 2024-01-24T14:20:54.828 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-01-24 14:13:29.530912822 +0000 2024-01-24T14:20:54.828 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-01-24 14:13:29.530912822 +0000 2024-01-24T14:20:54.828 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: - 2024-01-24T14:20:54.828 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-24T14:20:54.899 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-01-24T14:20:54.899 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-01-24T14:20:54.899 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000238864 s, 2.1 MB/s 2024-01-24T14:20:54.900 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-24T14:20:54.961 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_2 2024-01-24T14:20:55.020 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-24T14:20:55.021 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-24T14:20:55.021 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 6h/6d Inode: 172947 Links: 1 2024-01-24T14:20:55.021 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-24T14:20:55.021 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2024-01-24T14:20:55.021 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2024-01-24 14:20:41.879880142 +0000 2024-01-24T14:20:55.021 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-01-24 14:13:29.881904466 +0000 2024-01-24T14:20:55.021 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-01-24 14:13:29.881904466 +0000 2024-01-24T14:20:55.021 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: - 2024-01-24T14:20:55.021 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-24T14:20:55.093 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-01-24T14:20:55.093 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-01-24T14:20:55.093 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000304272 s, 1.7 MB/s 2024-01-24T14:20:55.094 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-24T14:20:55.095 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jan 24 14:20:54 smithi194 systemd[1]: Starting Ceph mgr.z for 654b0c8e-bac3-11ee-95b1-87774f69a715... 2024-01-24T14:20:55.156 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_3 2024-01-24T14:20:55.217 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-24T14:20:55.218 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-24T14:20:55.218 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 6h/6d Inode: 177356 Links: 1 2024-01-24T14:20:55.218 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-24T14:20:55.218 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2024-01-24T14:20:55.218 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2024-01-24 14:20:41.879880142 +0000 2024-01-24T14:20:55.218 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-01-24 14:13:30.246895778 +0000 2024-01-24T14:20:55.218 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-01-24 14:13:30.246895778 +0000 2024-01-24T14:20:55.218 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: - 2024-01-24T14:20:55.219 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-24T14:20:55.289 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-01-24T14:20:55.289 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-01-24T14:20:55.289 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000219626 s, 2.3 MB/s 2024-01-24T14:20:55.290 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-24T14:20:55.352 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_4 2024-01-24T14:20:55.414 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-24T14:20:55.415 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-24T14:20:55.415 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 6h/6d Inode: 177394 Links: 1 2024-01-24T14:20:55.415 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-24T14:20:55.415 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2024-01-24T14:20:55.415 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2024-01-24 14:20:41.879880142 +0000 2024-01-24T14:20:55.415 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-01-24 14:13:30.590887588 +0000 2024-01-24T14:20:55.415 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-01-24 14:13:30.590887588 +0000 2024-01-24T14:20:55.415 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: - 2024-01-24T14:20:55.415 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-24T14:20:55.486 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-01-24T14:20:55.487 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-01-24T14:20:55.487 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000257044 s, 2.0 MB/s 2024-01-24T14:20:55.488 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-24T14:20:55.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:55 smithi123 ceph-mon[44966]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi019=x;smithi123=y;smithi194=z", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:20:55.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:55 smithi123 ceph-mon[44966]: Saving service mgr spec with placement smithi019=x;smithi123=y;smithi194=z;count:3 2024-01-24T14:20:55.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:55 smithi123 ceph-mon[44966]: Deploying daemon mgr.z on smithi194 2024-01-24T14:20:55.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:55 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:55.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:55 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:55.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:55 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:55.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:55 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-24T14:20:55.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:55 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-01-24T14:20:55.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:55 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-24T14:20:55.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:55 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:20:55.551 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-01-24T14:20:55.551 DEBUG:teuthology.orchestra.run.smithi194:> dd if=/scratch_devs of=/dev/stdout 2024-01-24T14:20:55.569 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-24T14:20:55.569 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_1 2024-01-24T14:20:55.594 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jan 24 14:20:55 smithi194 ceph-mgr[47340]: -- 172.21.15.194:0/1851157055 <== mon.1 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5603957271e0 con 0x5603956fd000 2024-01-24T14:20:55.594 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jan 24 14:20:55 smithi194 bash[47302]: 084d3d4cebfd9c84e493d07eb66fd598cc1c47adb5a15cdf9bb53a07b5c1bc29 2024-01-24T14:20:55.594 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jan 24 14:20:55 smithi194 systemd[1]: Started Ceph mgr.z for 654b0c8e-bac3-11ee-95b1-87774f69a715. 2024-01-24T14:20:55.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:55 smithi194 ceph-mon[45450]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi019=x;smithi123=y;smithi194=z", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:20:55.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:55 smithi194 ceph-mon[45450]: Saving service mgr spec with placement smithi019=x;smithi123=y;smithi194=z;count:3 2024-01-24T14:20:55.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:55 smithi194 ceph-mon[45450]: Deploying daemon mgr.z on smithi194 2024-01-24T14:20:55.595 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:55 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:55.595 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:55 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:55.595 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:55 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:55.595 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:55 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-24T14:20:55.595 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:55 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-01-24T14:20:55.595 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:55 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-24T14:20:55.595 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:55 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:20:55.612 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-24T14:20:55.612 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-24T14:20:55.612 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 6h/6d Inode: 177341 Links: 1 2024-01-24T14:20:55.612 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-24T14:20:55.612 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-01-24T14:20:55.612 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-01-24 14:20:54.798389922 +0000 2024-01-24T14:20:55.612 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-01-24 14:13:20.013351145 +0000 2024-01-24T14:20:55.612 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-01-24 14:13:20.013351145 +0000 2024-01-24T14:20:55.612 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: - 2024-01-24T14:20:55.613 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-24T14:20:55.679 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jan 24 14:20:55 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-z[47317]: 2024-01-24T14:20:55.632+0000 7f2cb58aa200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-24T14:20:55.682 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-01-24T14:20:55.682 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-01-24T14:20:55.682 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000218069 s, 2.3 MB/s 2024-01-24T14:20:55.684 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-24T14:20:55.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:55 smithi019 ceph-mon[40663]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi019=x;smithi123=y;smithi194=z", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:20:55.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:55 smithi019 ceph-mon[40663]: Saving service mgr spec with placement smithi019=x;smithi123=y;smithi194=z;count:3 2024-01-24T14:20:55.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:55 smithi019 ceph-mon[40663]: Deploying daemon mgr.z on smithi194 2024-01-24T14:20:55.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:55 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:55.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:55 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:55.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:55 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:55.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:55 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-24T14:20:55.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:55 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-01-24T14:20:55.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:55 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-24T14:20:55.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:55 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:20:55.741 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_2 2024-01-24T14:20:55.799 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-24T14:20:55.799 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-24T14:20:55.799 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 6h/6d Inode: 171749 Links: 1 2024-01-24T14:20:55.799 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-24T14:20:55.800 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-01-24T14:20:55.800 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-01-24 14:20:54.799389901 +0000 2024-01-24T14:20:55.800 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-01-24 14:13:20.360343990 +0000 2024-01-24T14:20:55.800 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-01-24 14:13:20.360343990 +0000 2024-01-24T14:20:55.800 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: - 2024-01-24T14:20:55.800 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-24T14:20:55.866 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-01-24T14:20:55.866 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-01-24T14:20:55.866 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000242868 s, 2.1 MB/s 2024-01-24T14:20:55.867 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-24T14:20:55.910 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jan 24 14:20:55 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-z[47317]: 2024-01-24T14:20:55.729+0000 7f2cb58aa200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-24T14:20:55.910 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jan 24 14:20:55 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-z[47317]: 2024-01-24T14:20:55.820+0000 7f2cb58aa200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-24T14:20:55.910 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jan 24 14:20:55 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-z[47317]: 2024-01-24T14:20:55.910+0000 7f2cb58aa200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-24T14:20:55.925 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_3 2024-01-24T14:20:55.982 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-24T14:20:55.982 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-24T14:20:55.982 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 6h/6d Inode: 177425 Links: 1 2024-01-24T14:20:55.982 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-24T14:20:55.982 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-01-24T14:20:55.983 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-01-24 14:20:54.799389901 +0000 2024-01-24T14:20:55.983 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-01-24 14:13:20.709336794 +0000 2024-01-24T14:20:55.983 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-01-24 14:13:20.709336794 +0000 2024-01-24T14:20:55.983 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: - 2024-01-24T14:20:55.983 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-24T14:20:56.048 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-01-24T14:20:56.049 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-01-24T14:20:56.049 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000179179 s, 2.9 MB/s 2024-01-24T14:20:56.050 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-24T14:20:56.108 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_4 2024-01-24T14:20:56.165 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-24T14:20:56.166 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-24T14:20:56.166 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 6h/6d Inode: 175534 Links: 1 2024-01-24T14:20:56.166 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-24T14:20:56.166 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-01-24T14:20:56.166 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-01-24 14:20:54.799389901 +0000 2024-01-24T14:20:56.166 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-01-24 14:13:21.040329969 +0000 2024-01-24T14:20:56.166 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-01-24 14:13:21.040329969 +0000 2024-01-24T14:20:56.166 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: - 2024-01-24T14:20:56.166 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-24T14:20:56.233 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-01-24T14:20:56.233 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-01-24T14:20:56.233 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000217255 s, 2.4 MB/s 2024-01-24T14:20:56.234 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-24T14:20:56.292 INFO:tasks.cephadm:Deploying osd.0 on smithi019 with /dev/vg_nvme/lv_4... 2024-01-24T14:20:56.292 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-01-24T14:20:56.540 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/mon.a/config 2024-01-24T14:20:56.650 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:56 smithi019 ceph-mon[40663]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:56.650 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:56 smithi019 ceph-mon[40663]: Deploying daemon mgr.y on smithi123 2024-01-24T14:20:56.787 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:56 smithi194 ceph-mon[45450]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:56.787 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:56 smithi194 ceph-mon[45450]: Deploying daemon mgr.y on smithi123 2024-01-24T14:20:56.788 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jan 24 14:20:56 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-z[47317]: 2024-01-24T14:20:56.553+0000 7f2cb58aa200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-24T14:20:56.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:56 smithi123 ceph-mon[44966]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:56.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:56 smithi123 ceph-mon[44966]: Deploying daemon mgr.y on smithi123 2024-01-24T14:20:56.800 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Jan 24 14:20:56 smithi123 systemd[1]: Starting Ceph mgr.y for 654b0c8e-bac3-11ee-95b1-87774f69a715... 2024-01-24T14:20:57.094 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jan 24 14:20:56 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-z[47317]: 2024-01-24T14:20:56.787+0000 7f2cb58aa200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-24T14:20:57.094 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jan 24 14:20:56 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-z[47317]: 2024-01-24T14:20:56.871+0000 7f2cb58aa200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-24T14:20:57.094 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jan 24 14:20:56 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-z[47317]: 2024-01-24T14:20:56.978+0000 7f2cb58aa200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-24T14:20:57.299 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Jan 24 14:20:56 smithi123 ceph-mgr[46482]: -- 172.21.15.123:0/3678184895 <== mon.2 v2:172.21.15.123:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f4a85b91e0 con 0x55f4a858f000 2024-01-24T14:20:57.299 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Jan 24 14:20:56 smithi123 bash[46443]: 989cab7dd00be579a8db1c1efa494eb8bbfa377089076be257544be3d5ef8594 2024-01-24T14:20:57.299 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Jan 24 14:20:56 smithi123 systemd[1]: Started Ceph mgr.y for 654b0c8e-bac3-11ee-95b1-87774f69a715. 2024-01-24T14:20:57.443 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Jan 24 14:20:57 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-y[46459]: 2024-01-24T14:20:57.345+0000 7fa79a21b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-24T14:20:57.594 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jan 24 14:20:57 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-z[47317]: 2024-01-24T14:20:57.138+0000 7f2cb58aa200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-24T14:20:57.594 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jan 24 14:20:57 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-z[47317]: 2024-01-24T14:20:57.269+0000 7f2cb58aa200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-24T14:20:57.594 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jan 24 14:20:57 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-z[47317]: 2024-01-24T14:20:57.363+0000 7f2cb58aa200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-24T14:20:57.641 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Jan 24 14:20:57 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-y[46459]: 2024-01-24T14:20:57.442+0000 7fa79a21b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-24T14:20:57.641 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Jan 24 14:20:57 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-y[46459]: 2024-01-24T14:20:57.538+0000 7fa79a21b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-24T14:20:57.641 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Jan 24 14:20:57 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-y[46459]: 2024-01-24T14:20:57.640+0000 7fa79a21b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-24T14:20:57.889 INFO:teuthology.orchestra.run.smithi019.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-01-24T14:20:57.889 INFO:teuthology.orchestra.run.smithi019.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-01-24T14:20:57.889 INFO:teuthology.orchestra.run.smithi019.stderr: stderr: 10+0 records in 2024-01-24T14:20:57.889 INFO:teuthology.orchestra.run.smithi019.stderr:10+0 records out 2024-01-24T14:20:57.889 INFO:teuthology.orchestra.run.smithi019.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0182536 s, 574 MB/s 2024-01-24T14:20:57.889 INFO:teuthology.orchestra.run.smithi019.stderr:--> Zapping successful for: 2024-01-24T14:20:58.060 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jan 24 14:20:57 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-z[47317]: 2024-01-24T14:20:57.780+0000 7f2cb58aa200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-24T14:20:58.188 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:58 smithi019 ceph-mon[40663]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:58.188 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:58 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:58.188 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:58 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:58.188 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:58 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:58.188 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:58 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:58.188 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:58 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:20:58.188 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:58 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:58.188 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:58 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:58.188 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:58 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:58.282 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 -- ceph orch daemon add osd smithi019:vg_nvme/lv_4 2024-01-24T14:20:58.344 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jan 24 14:20:58 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-z[47317]: 2024-01-24T14:20:58.072+0000 7f2cb58aa200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-24T14:20:58.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:58 smithi194 ceph-mon[45450]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:58.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:58 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:58.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:58 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:58.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:58 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:58.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:58 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:58.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:58 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:20:58.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:58 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:58.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:58 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:58.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:58 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:58.521 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Jan 24 14:20:58 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-y[46459]: 2024-01-24T14:20:58.286+0000 7fa79a21b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-24T14:20:58.521 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:58 smithi123 ceph-mon[44966]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:20:58.522 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:58 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:58.522 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:58 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:58.522 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:58 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:58.522 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:58 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:58.522 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:58 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:20:58.522 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:58 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:58.522 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:58 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:58.522 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:58 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:58.543 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/mon.a/config 2024-01-24T14:20:58.799 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Jan 24 14:20:58 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-y[46459]: 2024-01-24T14:20:58.521+0000 7fa79a21b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-24T14:20:58.799 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Jan 24 14:20:58 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-y[46459]: 2024-01-24T14:20:58.605+0000 7fa79a21b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-24T14:20:58.799 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Jan 24 14:20:58 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-y[46459]: 2024-01-24T14:20:58.712+0000 7fa79a21b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-24T14:20:58.843 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jan 24 14:20:58 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-z[47317]: 2024-01-24T14:20:58.357+0000 7f2cb58aa200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-24T14:20:58.844 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jan 24 14:20:58 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-z[47317]: 2024-01-24T14:20:58.449+0000 7f2cb58aa200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-24T14:20:59.127 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:59 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:59.127 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:59 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:59.127 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:59 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:20:59.128 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:59 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:20:59.128 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:59 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:59.128 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:59 smithi019 ceph-mon[40663]: Reconfiguring mgr.x (unknown last config time)... 2024-01-24T14:20:59.128 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:59 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-24T14:20:59.128 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:59 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-24T14:20:59.128 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:59 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:20:59.128 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:20:59 smithi019 ceph-mon[40663]: Reconfiguring daemon mgr.x on smithi019 2024-01-24T14:20:59.299 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Jan 24 14:20:58 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-y[46459]: 2024-01-24T14:20:58.867+0000 7fa79a21b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-24T14:20:59.299 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Jan 24 14:20:58 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-y[46459]: 2024-01-24T14:20:58.992+0000 7fa79a21b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-24T14:20:59.299 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Jan 24 14:20:59 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-y[46459]: 2024-01-24T14:20:59.076+0000 7fa79a21b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-24T14:20:59.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:59 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:59.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:59 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:59.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:59 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:20:59.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:59 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:20:59.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:59 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:59.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:59 smithi123 ceph-mon[44966]: Reconfiguring mgr.x (unknown last config time)... 2024-01-24T14:20:59.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:59 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-24T14:20:59.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:59 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-24T14:20:59.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:59 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:20:59.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:20:59 smithi123 ceph-mon[44966]: Reconfiguring daemon mgr.x on smithi019 2024-01-24T14:20:59.344 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jan 24 14:20:59 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-z[47317]: 2024-01-24T14:20:59.196+0000 7f2cb58aa200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-24T14:20:59.344 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jan 24 14:20:59 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-z[47317]: 2024-01-24T14:20:59.289+0000 7f2cb58aa200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-24T14:20:59.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:59 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:59.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:59 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:59.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:59 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:20:59.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:59 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:20:59.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:59 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:20:59.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:59 smithi194 ceph-mon[45450]: Reconfiguring mgr.x (unknown last config time)... 2024-01-24T14:20:59.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:59 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-01-24T14:20:59.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:59 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-01-24T14:20:59.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:59 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:20:59.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:20:59 smithi194 ceph-mon[45450]: Reconfiguring daemon mgr.x on smithi019 2024-01-24T14:20:59.710 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Jan 24 14:20:59 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-y[46459]: 2024-01-24T14:20:59.444+0000 7fa79a21b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-24T14:20:59.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:59.825+0000 7ff0dbbb3700 1 -- 172.21.15.19:0/3784859862 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff0d4104250 msgr2=0x7ff0d4106640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:59.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:59.825+0000 7ff0dbbb3700 1 --2- 172.21.15.19:0/3784859862 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff0d4104250 0x7ff0d4106640 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff0c4009990 tx=0x7ff0c4009ca0 comp rx=0 tx=0).stop 2024-01-24T14:20:59.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:59.825+0000 7ff0dbbb3700 1 -- 172.21.15.19:0/3784859862 shutdown_connections 2024-01-24T14:20:59.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:59.825+0000 7ff0dbbb3700 1 --2- 172.21.15.19:0/3784859862 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0d4106b80 0x7ff0d4108f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:59.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:59.825+0000 7ff0dbbb3700 1 --2- 172.21.15.19:0/3784859862 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff0d4101920 0x7ff0d4103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:59.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:59.825+0000 7ff0dbbb3700 1 --2- 172.21.15.19:0/3784859862 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff0d4104250 0x7ff0d4106640 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:59.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:59.825+0000 7ff0dbbb3700 1 -- 172.21.15.19:0/3784859862 >> 172.21.15.19:0/3784859862 conn(0x7ff0d40fb250 msgr2=0x7ff0d40fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:20:59.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:59.825+0000 7ff0dbbb3700 1 -- 172.21.15.19:0/3784859862 shutdown_connections 2024-01-24T14:20:59.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:59.825+0000 7ff0dbbb3700 1 -- 172.21.15.19:0/3784859862 wait complete. 2024-01-24T14:20:59.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:59.826+0000 7ff0dbbb3700 1 Processor -- start 2024-01-24T14:20:59.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:59.826+0000 7ff0dbbb3700 1 -- start start 2024-01-24T14:20:59.831 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:59.826+0000 7ff0dbbb3700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0d4101920 0x7ff0d4196a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:59.831 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:59.826+0000 7ff0dbbb3700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff0d4104250 0x7ff0d4196fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:59.831 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:59.826+0000 7ff0dbbb3700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff0d4106b80 0x7ff0d419e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:59.831 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:59.826+0000 7ff0dbbb3700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d419e600 con 0x7ff0d4104250 2024-01-24T14:20:59.831 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:59.826+0000 7ff0dbbb3700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d419e740 con 0x7ff0d4101920 2024-01-24T14:20:59.831 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:59.826+0000 7ff0dbbb3700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d419e8d0 con 0x7ff0d4106b80 2024-01-24T14:20:59.831 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:59.826+0000 7ff0d914e700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff0d4104250 0x7ff0d4196fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:59.831 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:59.826+0000 7ff0d994f700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0d4101920 0x7ff0d4196a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:59.831 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:59.826+0000 7ff0d914e700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff0d4104250 0x7ff0d4196fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:35620/0 (socket says 172.21.15.19:35620) 2024-01-24T14:20:59.831 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:59.826+0000 7ff0d914e700 1 -- 172.21.15.19:0/4248470744 learned_addr learned my addr 172.21.15.19:0/4248470744 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:20:59.831 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:59.826+0000 7ff0da150700 1 --2- 172.21.15.19:0/4248470744 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff0d4106b80 0x7ff0d419e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:59.832 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:59.827+0000 7ff0d994f700 1 -- 172.21.15.19:0/4248470744 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff0d4106b80 msgr2=0x7ff0d419e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:59.832 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:59.827+0000 7ff0d994f700 1 --2- 172.21.15.19:0/4248470744 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff0d4106b80 0x7ff0d419e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:59.832 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:59.827+0000 7ff0d994f700 1 -- 172.21.15.19:0/4248470744 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff0d4104250 msgr2=0x7ff0d4196fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:20:59.832 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:59.827+0000 7ff0d994f700 1 --2- 172.21.15.19:0/4248470744 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff0d4104250 0x7ff0d4196fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:20:59.832 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:59.827+0000 7ff0d994f700 1 -- 172.21.15.19:0/4248470744 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c40095f0 con 0x7ff0d4101920 2024-01-24T14:20:59.832 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:59.827+0000 7ff0d914e700 1 --2- 172.21.15.19:0/4248470744 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff0d4104250 0x7ff0d4196fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T14:20:59.832 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:59.827+0000 7ff0d994f700 1 --2- 172.21.15.19:0/4248470744 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0d4101920 0x7ff0d4196a70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff0d000e9f0 tx=0x7ff0d000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:20:59.832 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:59.827+0000 7ff0caffd700 1 -- 172.21.15.19:0/4248470744 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d000c9c0 con 0x7ff0d4101920 2024-01-24T14:20:59.832 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:59.827+0000 7ff0caffd700 1 -- 172.21.15.19:0/4248470744 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff0d0010430 con 0x7ff0d4101920 2024-01-24T14:20:59.832 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:59.827+0000 7ff0dbbb3700 1 -- 172.21.15.19:0/4248470744 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0d419ebb0 con 0x7ff0d4101920 2024-01-24T14:20:59.832 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:59.827+0000 7ff0caffd700 1 -- 172.21.15.19:0/4248470744 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d0018680 con 0x7ff0d4101920 2024-01-24T14:20:59.832 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:59.827+0000 7ff0dbbb3700 1 -- 172.21.15.19:0/4248470744 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0d419f100 con 0x7ff0d4101920 2024-01-24T14:20:59.832 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:59.828+0000 7ff0caffd700 1 -- 172.21.15.19:0/4248470744 <== mon.2 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 13) v1 ==== 45324+0+0 (secure 0 0 0) 0x7ff0d00187e0 con 0x7ff0d4101920 2024-01-24T14:20:59.833 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:59.828+0000 7ff0dbbb3700 1 -- 172.21.15.19:0/4248470744 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0d410c600 con 0x7ff0d4101920 2024-01-24T14:20:59.833 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:59.828+0000 7ff0caffd700 1 --2- 172.21.15.19:0/4248470744 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7ff0c00385d0 0x7ff0c003aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:20:59.833 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:59.828+0000 7ff0caffd700 1 -- 172.21.15.19:0/4248470744 <== mon.2 v2:172.21.15.123:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff0d0014070 con 0x7ff0d4101920 2024-01-24T14:20:59.833 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:59.828+0000 7ff0d914e700 1 --2- 172.21.15.19:0/4248470744 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7ff0c00385d0 0x7ff0c003aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:20:59.833 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:59.829+0000 7ff0d914e700 1 --2- 172.21.15.19:0/4248470744 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7ff0c00385d0 0x7ff0c003aa90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff0c400b2e0 tx=0x7ff0c4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:20:59.834 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:20:59.834+0000 7ff0caffd700 1 -- 172.21.15.19:0/4248470744 <== mon.2 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7ff0d001fd50 con 0x7ff0d4101920 2024-01-24T14:20:59.843 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jan 24 14:20:59 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-z[47317]: 2024-01-24T14:20:59.435+0000 7f2cb58aa200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-24T14:20:59.991 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Jan 24 14:20:59 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-y[46459]: 2024-01-24T14:20:59.709+0000 7fa79a21b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-24T14:21:00.051 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:00.050+0000 7ff0dbbb3700 1 -- 172.21.15.19:0/4248470744 --> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7ff0d4103480 con 0x7ff0c00385d0 2024-01-24T14:21:00.299 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Jan 24 14:20:59 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-y[46459]: 2024-01-24T14:20:59.991+0000 7fa79a21b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-24T14:21:00.299 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Jan 24 14:21:00 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-y[46459]: 2024-01-24T14:21:00.083+0000 7fa79a21b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-24T14:21:00.344 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jan 24 14:21:00 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-z[47317]: 2024-01-24T14:21:00.062+0000 7f2cb58aa200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-24T14:21:00.344 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jan 24 14:21:00 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-z[47317]: 2024-01-24T14:21:00.240+0000 7f2cb58aa200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-24T14:21:00.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:00 smithi019 ceph-mon[40663]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:21:00.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:00 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:00.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:00 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:00.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:00 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:21:00.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:00 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:21:00.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:00 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:21:00.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:00 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:00.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:00 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T14:21:00.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:00 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T14:21:00.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:00 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:21:00.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:00 smithi123 ceph-mon[44966]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:21:00.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:00 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:00.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:00 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:00.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:00 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:21:00.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:00 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:21:00.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:00 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:21:00.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:00 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:00.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:00 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T14:21:00.801 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:00 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T14:21:00.801 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:00 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:21:00.844 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jan 24 14:21:00 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-z[47317]: 2024-01-24T14:21:00.431+0000 7f2cb58aa200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-24T14:21:00.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:00 smithi194 ceph-mon[45450]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:21:00.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:00 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:00.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:00 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:00.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:00 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:21:00.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:00 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:21:00.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:00 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:21:00.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:00 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:00.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:00 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T14:21:00.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:00 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T14:21:00.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:00 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:21:01.299 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Jan 24 14:21:00 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-y[46459]: 2024-01-24T14:21:00.827+0000 7fa79a21b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-24T14:21:01.299 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Jan 24 14:21:00 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-y[46459]: 2024-01-24T14:21:00.921+0000 7fa79a21b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-24T14:21:01.299 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Jan 24 14:21:01 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-y[46459]: 2024-01-24T14:21:01.066+0000 7fa79a21b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-24T14:21:01.485 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:01 smithi019 ceph-mon[40663]: from='client.24122 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:21:01.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:01 smithi194 ceph-mon[45450]: from='client.24122 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:21:01.594 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jan 24 14:21:01 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-z[47317]: 2024-01-24T14:21:01.342+0000 7f2cb58aa200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-24T14:21:01.687 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:01 smithi123 ceph-mon[44966]: from='client.24122 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:21:01.883 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jan 24 14:21:01 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-z[47317]: 2024-01-24T14:21:01.607+0000 7f2cb58aa200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-24T14:21:01.883 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jan 24 14:21:01 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-z[47317]: 2024-01-24T14:21:01.693+0000 7f2cb58aa200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-24T14:21:02.049 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Jan 24 14:21:01 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-y[46459]: 2024-01-24T14:21:01.686+0000 7fa79a21b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-24T14:21:02.049 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Jan 24 14:21:01 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-y[46459]: 2024-01-24T14:21:01.866+0000 7fa79a21b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-24T14:21:02.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:02.374+0000 7ff0caffd700 1 -- 172.21.15.19:0/4248470744 <== mon.2 v2:172.21.15.123:3300/0 7 ==== mgrmap(e 14) v1 ==== 90169+0+0 (secure 0 0 0) 0x7ff0d0010860 con 0x7ff0d4101920 2024-01-24T14:21:02.378 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Jan 24 14:21:02 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-y[46459]: 2024-01-24T14:21:02.056+0000 7fa79a21b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-24T14:21:02.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:02 smithi019 ceph-mon[40663]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:21:02.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:02 smithi019 ceph-mon[40663]: Standby manager daemon z started 2024-01-24T14:21:02.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:02 smithi019 ceph-mon[40663]: from='mgr.? 172.21.15.194:0/465764371' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-01-24T14:21:02.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:02 smithi019 ceph-mon[40663]: from='mgr.? 172.21.15.194:0/465764371' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-01-24T14:21:02.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:02 smithi019 ceph-mon[40663]: from='mgr.? 172.21.15.194:0/465764371' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-01-24T14:21:02.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:02 smithi019 ceph-mon[40663]: from='mgr.? 172.21.15.194:0/465764371' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-01-24T14:21:02.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:02 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/2036696412' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "81380ec6-8e19-4eb4-8158-8fafaed8d7e9"}]: dispatch 2024-01-24T14:21:02.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:02 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/2036696412' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "81380ec6-8e19-4eb4-8158-8fafaed8d7e9"}]': finished 2024-01-24T14:21:02.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:02 smithi019 ceph-mon[40663]: osdmap e5: 1 total, 0 up, 1 in 2024-01-24T14:21:02.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:02 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-24T14:21:02.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:02 smithi123 ceph-mon[44966]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:21:02.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:02 smithi123 ceph-mon[44966]: Standby manager daemon z started 2024-01-24T14:21:02.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:02 smithi123 ceph-mon[44966]: from='mgr.? 172.21.15.194:0/465764371' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-01-24T14:21:02.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:02 smithi123 ceph-mon[44966]: from='mgr.? 172.21.15.194:0/465764371' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-01-24T14:21:02.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:02 smithi123 ceph-mon[44966]: from='mgr.? 172.21.15.194:0/465764371' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-01-24T14:21:02.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:02 smithi123 ceph-mon[44966]: from='mgr.? 172.21.15.194:0/465764371' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-01-24T14:21:02.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:02 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/2036696412' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "81380ec6-8e19-4eb4-8158-8fafaed8d7e9"}]: dispatch 2024-01-24T14:21:02.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:02 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/2036696412' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "81380ec6-8e19-4eb4-8158-8fafaed8d7e9"}]': finished 2024-01-24T14:21:02.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:02 smithi123 ceph-mon[44966]: osdmap e5: 1 total, 0 up, 1 in 2024-01-24T14:21:02.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:02 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-24T14:21:02.844 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jan 24 14:21:01 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-z[47317]: 2024-01-24T14:21:01.883+0000 7f2cb58aa200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-24T14:21:02.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:02 smithi194 ceph-mon[45450]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:21:02.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:02 smithi194 ceph-mon[45450]: Standby manager daemon z started 2024-01-24T14:21:02.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:02 smithi194 ceph-mon[45450]: from='mgr.? 172.21.15.194:0/465764371' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-01-24T14:21:02.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:02 smithi194 ceph-mon[45450]: from='mgr.? 172.21.15.194:0/465764371' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-01-24T14:21:02.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:02 smithi194 ceph-mon[45450]: from='mgr.? 172.21.15.194:0/465764371' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-01-24T14:21:02.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:02 smithi194 ceph-mon[45450]: from='mgr.? 172.21.15.194:0/465764371' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-01-24T14:21:02.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:02 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/2036696412' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "81380ec6-8e19-4eb4-8158-8fafaed8d7e9"}]: dispatch 2024-01-24T14:21:02.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:02 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/2036696412' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "81380ec6-8e19-4eb4-8158-8fafaed8d7e9"}]': finished 2024-01-24T14:21:02.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:02 smithi194 ceph-mon[45450]: osdmap e5: 1 total, 0 up, 1 in 2024-01-24T14:21:02.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:02 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-24T14:21:03.231 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Jan 24 14:21:02 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-y[46459]: 2024-01-24T14:21:02.966+0000 7fa79a21b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-24T14:21:03.505 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:03 smithi123 ceph-mon[44966]: mgrmap e14: x(active, since 59s), standbys: z 2024-01-24T14:21:03.505 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:03 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-01-24T14:21:03.505 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:03 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/185308993' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T14:21:03.505 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Jan 24 14:21:03 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-y[46459]: 2024-01-24T14:21:03.230+0000 7fa79a21b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-24T14:21:03.506 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Jan 24 14:21:03 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-y[46459]: 2024-01-24T14:21:03.316+0000 7fa79a21b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-24T14:21:03.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:03 smithi019 ceph-mon[40663]: mgrmap e14: x(active, since 59s), standbys: z 2024-01-24T14:21:03.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:03 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-01-24T14:21:03.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:03 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/185308993' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T14:21:03.799 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Jan 24 14:21:03 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-y[46459]: 2024-01-24T14:21:03.504+0000 7fa79a21b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-24T14:21:03.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:03 smithi194 ceph-mon[45450]: mgrmap e14: x(active, since 59s), standbys: z 2024-01-24T14:21:03.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:03 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-01-24T14:21:03.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:03 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/185308993' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T14:21:04.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:04.399+0000 7ff0caffd700 1 -- 172.21.15.19:0/4248470744 <== mon.2 v2:172.21.15.123:3300/0 8 ==== mgrmap(e 15) v1 ==== 135014+0+0 (secure 0 0 0) 0x7ff0d0017750 con 0x7ff0d4101920 2024-01-24T14:21:04.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:04 smithi019 ceph-mon[40663]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:21:04.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:04 smithi019 ceph-mon[40663]: Standby manager daemon y started 2024-01-24T14:21:04.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:04 smithi019 ceph-mon[40663]: from='mgr.? 172.21.15.123:0/1410806939' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-01-24T14:21:04.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:04 smithi019 ceph-mon[40663]: from='mgr.? 172.21.15.123:0/1410806939' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-01-24T14:21:04.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:04 smithi019 ceph-mon[40663]: from='mgr.? 172.21.15.123:0/1410806939' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-01-24T14:21:04.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:04 smithi019 ceph-mon[40663]: from='mgr.? 172.21.15.123:0/1410806939' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-01-24T14:21:04.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:04 smithi123 ceph-mon[44966]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:21:04.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:04 smithi123 ceph-mon[44966]: Standby manager daemon y started 2024-01-24T14:21:04.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:04 smithi123 ceph-mon[44966]: from='mgr.? 172.21.15.123:0/1410806939' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-01-24T14:21:04.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:04 smithi123 ceph-mon[44966]: from='mgr.? 172.21.15.123:0/1410806939' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-01-24T14:21:04.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:04 smithi123 ceph-mon[44966]: from='mgr.? 172.21.15.123:0/1410806939' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-01-24T14:21:04.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:04 smithi123 ceph-mon[44966]: from='mgr.? 172.21.15.123:0/1410806939' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-01-24T14:21:04.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:04 smithi194 ceph-mon[45450]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:21:04.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:04 smithi194 ceph-mon[45450]: Standby manager daemon y started 2024-01-24T14:21:04.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:04 smithi194 ceph-mon[45450]: from='mgr.? 172.21.15.123:0/1410806939' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-01-24T14:21:04.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:04 smithi194 ceph-mon[45450]: from='mgr.? 172.21.15.123:0/1410806939' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-01-24T14:21:04.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:04 smithi194 ceph-mon[45450]: from='mgr.? 172.21.15.123:0/1410806939' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-01-24T14:21:04.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:04 smithi194 ceph-mon[45450]: from='mgr.? 172.21.15.123:0/1410806939' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-01-24T14:21:05.713 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:05 smithi019 ceph-mon[40663]: mgrmap e15: x(active, since 61s), standbys: z, y 2024-01-24T14:21:05.713 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:05 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-01-24T14:21:05.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:05 smithi123 ceph-mon[44966]: mgrmap e15: x(active, since 61s), standbys: z, y 2024-01-24T14:21:05.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:05 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-01-24T14:21:05.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:05 smithi194 ceph-mon[45450]: mgrmap e15: x(active, since 61s), standbys: z, y 2024-01-24T14:21:05.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:05 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-01-24T14:21:06.664 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:06 smithi019 ceph-mon[40663]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:21:06.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:06 smithi123 ceph-mon[44966]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:21:06.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:06 smithi194 ceph-mon[45450]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:21:08.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:08 smithi019 ceph-mon[40663]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:21:08.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:08 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-01-24T14:21:08.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:08 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:21:08.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:08 smithi123 ceph-mon[44966]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:21:08.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:08 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-01-24T14:21:08.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:08 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:21:08.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:08 smithi194 ceph-mon[45450]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:21:08.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:08 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-01-24T14:21:08.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:08 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:21:09.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:09 smithi019 ceph-mon[40663]: Deploying daemon osd.0 on smithi019 2024-01-24T14:21:09.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:09 smithi123 ceph-mon[44966]: Deploying daemon osd.0 on smithi019 2024-01-24T14:21:09.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:09 smithi194 ceph-mon[45450]: Deploying daemon osd.0 on smithi019 2024-01-24T14:21:10.458 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:10 smithi019 ceph-mon[40663]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:21:10.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:10 smithi123 ceph-mon[44966]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:21:10.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:10 smithi194 ceph-mon[45450]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:21:12.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:12 smithi019 ceph-mon[40663]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:21:12.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:12 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:12.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:12 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:12.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:12 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:21:12.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:12 smithi123 ceph-mon[44966]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:21:12.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:12 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:12.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:12 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:12.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:12 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:21:12.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:12 smithi194 ceph-mon[45450]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:21:12.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:12 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:12.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:12 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:12.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:12 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:21:14.401 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:14.399+0000 7ff0caffd700 1 -- 172.21.15.19:0/4248470744 <== mgr.14152 v2:172.21.15.19:6800/1634350318 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff0d4103480 con 0x7ff0c00385d0 2024-01-24T14:21:14.401 INFO:teuthology.orchestra.run.smithi019.stdout:Created osd(s) 0 on host 'smithi019' 2024-01-24T14:21:14.404 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:14.402+0000 7ff0dbbb3700 1 -- 172.21.15.19:0/4248470744 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7ff0c00385d0 msgr2=0x7ff0c003aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:21:14.404 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:14.402+0000 7ff0dbbb3700 1 --2- 172.21.15.19:0/4248470744 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7ff0c00385d0 0x7ff0c003aa90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff0c400b2e0 tx=0x7ff0c4005fb0 comp rx=0 tx=0).stop 2024-01-24T14:21:14.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:14.402+0000 7ff0dbbb3700 1 -- 172.21.15.19:0/4248470744 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0d4101920 msgr2=0x7ff0d4196a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:21:14.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:14.402+0000 7ff0dbbb3700 1 --2- 172.21.15.19:0/4248470744 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0d4101920 0x7ff0d4196a70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff0d000e9f0 tx=0x7ff0d000ed00 comp rx=0 tx=0).stop 2024-01-24T14:21:14.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:14.402+0000 7ff0dbbb3700 1 -- 172.21.15.19:0/4248470744 shutdown_connections 2024-01-24T14:21:14.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:14.402+0000 7ff0dbbb3700 1 --2- 172.21.15.19:0/4248470744 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff0d4106b80 0x7ff0d419e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:21:14.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:14.402+0000 7ff0dbbb3700 1 --2- 172.21.15.19:0/4248470744 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0d4101920 0x7ff0d4196a70 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:21:14.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:14.402+0000 7ff0dbbb3700 1 --2- 172.21.15.19:0/4248470744 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7ff0c00385d0 0x7ff0c003aa90 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:21:14.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:14.402+0000 7ff0dbbb3700 1 --2- 172.21.15.19:0/4248470744 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff0d4104250 0x7ff0d4196fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:21:14.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:14.402+0000 7ff0dbbb3700 1 -- 172.21.15.19:0/4248470744 >> 172.21.15.19:0/4248470744 conn(0x7ff0d40fb250 msgr2=0x7ff0d4102d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:21:14.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:14.403+0000 7ff0dbbb3700 1 -- 172.21.15.19:0/4248470744 shutdown_connections 2024-01-24T14:21:14.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:14.403+0000 7ff0dbbb3700 1 -- 172.21.15.19:0/4248470744 wait complete. 2024-01-24T14:21:14.618 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:14 smithi019 ceph-mon[40663]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:21:14.618 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:14 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:14.618 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:14 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:14.619 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:14 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:21:14.619 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:14 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:21:14.619 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:14 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:14.619 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:14 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:14.619 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:14 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:14.619 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:14 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:21:14.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:14 smithi123 ceph-mon[44966]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:21:14.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:14 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:14.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:14 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:14.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:14 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:21:14.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:14 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:21:14.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:14 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:14.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:14 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:14.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:14 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:14.801 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:14 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:21:14.813 DEBUG:teuthology.orchestra.run.smithi019:osd.0> sudo journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@osd.0.service 2024-01-24T14:21:14.816 INFO:tasks.cephadm:Deploying osd.1 on smithi123 with /dev/vg_nvme/lv_4... 2024-01-24T14:21:14.816 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-01-24T14:21:14.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:14 smithi194 ceph-mon[45450]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:21:14.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:14 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:14.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:14 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:14.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:14 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:21:14.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:14 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:21:14.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:14 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:14.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:14 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:14.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:14 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:14.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:14 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:21:14.854 INFO:journalctl@ceph.osd.0.smithi019.stdout:-- Logs begin at Wed 2024-01-24 14:03:52 UTC. -- 2024-01-24T14:21:15.048 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/mon.b/config 2024-01-24T14:21:16.334 INFO:teuthology.orchestra.run.smithi123.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-01-24T14:21:16.334 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-01-24T14:21:16.334 INFO:teuthology.orchestra.run.smithi123.stderr: stderr: 10+0 records in 2024-01-24T14:21:16.334 INFO:teuthology.orchestra.run.smithi123.stderr:10+0 records out 2024-01-24T14:21:16.334 INFO:teuthology.orchestra.run.smithi123.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0170939 s, 613 MB/s 2024-01-24T14:21:16.334 INFO:teuthology.orchestra.run.smithi123.stderr:--> Zapping successful for: 2024-01-24T14:21:16.657 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:16 smithi123 ceph-mon[44966]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:21:16.657 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:16 smithi123 ceph-mon[44966]: from='osd.0 [v2:172.21.15.19:6802/2456786579,v1:172.21.15.19:6803/2456786579]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-01-24T14:21:16.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:16 smithi019 ceph-mon[40663]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:21:16.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:16 smithi019 ceph-mon[40663]: from='osd.0 [v2:172.21.15.19:6802/2456786579,v1:172.21.15.19:6803/2456786579]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-01-24T14:21:16.736 INFO:journalctl@ceph.osd.0.smithi019.stdout:Jan 24 14:21:16 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-0[59108]: 2024-01-24T14:21:16.240+0000 7fe1e2bf1640 -1 osd.0 0 log_to_monitors true 2024-01-24T14:21:16.740 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 -- ceph orch daemon add osd smithi123:vg_nvme/lv_4 2024-01-24T14:21:16.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:16 smithi194 ceph-mon[45450]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:21:16.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:16 smithi194 ceph-mon[45450]: from='osd.0 [v2:172.21.15.19:6802/2456786579,v1:172.21.15.19:6803/2456786579]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-01-24T14:21:16.954 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/mon.b/config 2024-01-24T14:21:17.730 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:17 smithi123 ceph-mon[44966]: from='osd.0 [v2:172.21.15.19:6802/2456786579,v1:172.21.15.19:6803/2456786579]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-01-24T14:21:17.730 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:17 smithi123 ceph-mon[44966]: osdmap e6: 1 total, 0 up, 1 in 2024-01-24T14:21:17.730 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:17 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-24T14:21:17.730 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:17 smithi123 ceph-mon[44966]: from='osd.0 [v2:172.21.15.19:6802/2456786579,v1:172.21.15.19:6803/2456786579]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-01-24T14:21:17.730 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:17 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:17.730 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:17 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:17.730 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:17 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:21:17.730 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:17 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:17.730 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:17 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:21:17.730 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:17 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:21:17.730 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:17 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:17.735 INFO:journalctl@ceph.osd.0.smithi019.stdout:Jan 24 14:21:17 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-0[59108]: 2024-01-24T14:21:17.437+0000 7fe1d8256700 -1 osd.0 0 waiting for initial osdmap 2024-01-24T14:21:17.736 INFO:journalctl@ceph.osd.0.smithi019.stdout:Jan 24 14:21:17 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-0[59108]: 2024-01-24T14:21:17.449+0000 7fe1d0043700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-24T14:21:17.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:17 smithi019 ceph-mon[40663]: from='osd.0 [v2:172.21.15.19:6802/2456786579,v1:172.21.15.19:6803/2456786579]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-01-24T14:21:17.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:17 smithi019 ceph-mon[40663]: osdmap e6: 1 total, 0 up, 1 in 2024-01-24T14:21:17.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:17 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-24T14:21:17.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:17 smithi019 ceph-mon[40663]: from='osd.0 [v2:172.21.15.19:6802/2456786579,v1:172.21.15.19:6803/2456786579]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-01-24T14:21:17.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:17 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:17.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:17 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:17.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:17 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:21:17.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:17 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:17.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:17 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:21:17.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:17 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:21:17.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:17 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:17.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:17 smithi194 ceph-mon[45450]: from='osd.0 [v2:172.21.15.19:6802/2456786579,v1:172.21.15.19:6803/2456786579]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-01-24T14:21:17.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:17 smithi194 ceph-mon[45450]: osdmap e6: 1 total, 0 up, 1 in 2024-01-24T14:21:17.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:17 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-24T14:21:17.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:17 smithi194 ceph-mon[45450]: from='osd.0 [v2:172.21.15.19:6802/2456786579,v1:172.21.15.19:6803/2456786579]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-01-24T14:21:17.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:17 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:17.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:17 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:17.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:17 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:21:17.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:17 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:17.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:17 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:21:17.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:17 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:21:17.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:17 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:17.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:17.939+0000 7f82b80f8700 1 -- 172.21.15.123:0/3307732830 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82b0106ac0 msgr2=0x7f82b0108f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:21:17.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:17.939+0000 7f82b80f8700 1 --2- 172.21.15.123:0/3307732830 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82b0106ac0 0x7f82b0108f30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f82a8009f90 tx=0x7f82a800b330 comp rx=0 tx=0).stop 2024-01-24T14:21:17.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:17.939+0000 7f82b80f8700 1 -- 172.21.15.123:0/3307732830 shutdown_connections 2024-01-24T14:21:17.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:17.939+0000 7f82b80f8700 1 --2- 172.21.15.123:0/3307732830 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82b0106ac0 0x7f82b0108f30 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:21:17.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:17.939+0000 7f82b80f8700 1 --2- 172.21.15.123:0/3307732830 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f82b0101860 0x7f82b0103c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:21:17.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:17.939+0000 7f82b80f8700 1 --2- 172.21.15.123:0/3307732830 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f82b0104190 0x7f82b0106580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:21:17.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:17.939+0000 7f82b80f8700 1 -- 172.21.15.123:0/3307732830 >> 172.21.15.123:0/3307732830 conn(0x7f82b00fb250 msgr2=0x7f82b00fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:21:17.940 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:17.939+0000 7f82b80f8700 1 -- 172.21.15.123:0/3307732830 shutdown_connections 2024-01-24T14:21:17.941 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:17.939+0000 7f82b80f8700 1 -- 172.21.15.123:0/3307732830 wait complete. 2024-01-24T14:21:17.942 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:17.940+0000 7f82b80f8700 1 Processor -- start 2024-01-24T14:21:17.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:17.940+0000 7f82b80f8700 1 -- start start 2024-01-24T14:21:17.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:17.940+0000 7f82b80f8700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f82b0101860 0x7f82b0198db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:21:17.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:17.940+0000 7f82b80f8700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82b0104190 0x7f82b01992f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:21:17.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:17.940+0000 7f82b80f8700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f82b0106ac0 0x7f82b01a0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:21:17.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:17.940+0000 7f82b80f8700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b01a0900 con 0x7f82b0106ac0 2024-01-24T14:21:17.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:17.940+0000 7f82b80f8700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b01a0a40 con 0x7f82b0104190 2024-01-24T14:21:17.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:17.940+0000 7f82b80f8700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b01a0c00 con 0x7f82b0101860 2024-01-24T14:21:17.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:17.940+0000 7f82b5693700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82b0104190 0x7f82b01992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:21:17.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:17.940+0000 7f82b5693700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82b0104190 0x7f82b01992f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49962/0 (socket says 172.21.15.123:49962) 2024-01-24T14:21:17.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:17.940+0000 7f82b5693700 1 -- 172.21.15.123:0/506867187 learned_addr learned my addr 172.21.15.123:0/506867187 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-24T14:21:17.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:17.940+0000 7f82b6695700 1 --2- 172.21.15.123:0/506867187 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f82b0106ac0 0x7f82b01a0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:21:17.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:17.940+0000 7f82b5e94700 1 --2- 172.21.15.123:0/506867187 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f82b0101860 0x7f82b0198db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:21:17.943 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:17.941+0000 7f82b5693700 1 -- 172.21.15.123:0/506867187 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f82b0101860 msgr2=0x7f82b0198db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:21:17.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:17.941+0000 7f82b5693700 1 --2- 172.21.15.123:0/506867187 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f82b0101860 0x7f82b0198db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:21:17.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:17.941+0000 7f82b5693700 1 -- 172.21.15.123:0/506867187 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f82b0106ac0 msgr2=0x7f82b01a0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:21:17.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:17.941+0000 7f82b5693700 1 --2- 172.21.15.123:0/506867187 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f82b0106ac0 0x7f82b01a0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:21:17.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:17.941+0000 7f82b5693700 1 -- 172.21.15.123:0/506867187 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82a8009c40 con 0x7f82b0104190 2024-01-24T14:21:17.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:17.941+0000 7f82b5693700 1 --2- 172.21.15.123:0/506867187 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82b0104190 0x7f82b01992f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f82ac00e9a0 tx=0x7f82ac00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:21:17.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:17.941+0000 7f82a6ffd700 1 -- 172.21.15.123:0/506867187 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ac009790 con 0x7f82b0104190 2024-01-24T14:21:17.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:17.941+0000 7f82a6ffd700 1 -- 172.21.15.123:0/506867187 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f82ac00cbd0 con 0x7f82b0104190 2024-01-24T14:21:17.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:17.941+0000 7f82b80f8700 1 -- 172.21.15.123:0/506867187 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82b01a0ee0 con 0x7f82b0104190 2024-01-24T14:21:17.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:17.941+0000 7f82a6ffd700 1 -- 172.21.15.123:0/506867187 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ac018680 con 0x7f82b0104190 2024-01-24T14:21:17.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:17.941+0000 7f82b80f8700 1 -- 172.21.15.123:0/506867187 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82b01a1350 con 0x7f82b0104190 2024-01-24T14:21:17.944 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:17.942+0000 7f82b80f8700 1 -- 172.21.15.123:0/506867187 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82b0108640 con 0x7f82b0104190 2024-01-24T14:21:17.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:17.943+0000 7f82a6ffd700 1 -- 172.21.15.123:0/506867187 <== mon.2 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 15) v1 ==== 135014+0+0 (secure 0 0 0) 0x7f82ac00cd40 con 0x7f82b0104190 2024-01-24T14:21:17.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:17.943+0000 7f82a6ffd700 1 --2- 172.21.15.123:0/506867187 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f829c0a0510 0x7f829c0a29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:21:17.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:17.943+0000 7f82a6ffd700 1 -- 172.21.15.123:0/506867187 <== mon.2 v2:172.21.15.123:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1487+0+0 (secure 0 0 0) 0x7f82ac014070 con 0x7f82b0104190 2024-01-24T14:21:17.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:17.944+0000 7f82b5e94700 1 --2- 172.21.15.123:0/506867187 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f829c0a0510 0x7f829c0a29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:21:17.945 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:17.944+0000 7f82b5e94700 1 --2- 172.21.15.123:0/506867187 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f829c0a0510 0x7f829c0a29d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f82b019a290 tx=0x7f82a0009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:21:17.948 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:17.948+0000 7f82a6ffd700 1 -- 172.21.15.123:0/506867187 <== mon.2 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f82ac052120 con 0x7f82b0104190 2024-01-24T14:21:18.141 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:18.141+0000 7f82b80f8700 1 -- 172.21.15.123:0/506867187 --> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f82b0061180 con 0x7f829c0a0510 2024-01-24T14:21:18.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:18 smithi019 ceph-mon[40663]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:21:18.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:18 smithi019 ceph-mon[40663]: Detected new or changed devices on smithi019 2024-01-24T14:21:18.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:18 smithi019 ceph-mon[40663]: Adjusting osd_memory_target on smithi019 to 17118M 2024-01-24T14:21:18.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:18 smithi019 ceph-mon[40663]: from='osd.0 [v2:172.21.15.19:6802/2456786579,v1:172.21.15.19:6803/2456786579]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]': finished 2024-01-24T14:21:18.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:18 smithi019 ceph-mon[40663]: osdmap e7: 1 total, 0 up, 1 in 2024-01-24T14:21:18.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:18 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-24T14:21:18.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:18 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-24T14:21:18.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:18 smithi019 ceph-mon[40663]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:21:18.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:18 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T14:21:18.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:18 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T14:21:18.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:18 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:21:18.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:18 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-24T14:21:18.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:18 smithi123 ceph-mon[44966]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:21:18.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:18 smithi123 ceph-mon[44966]: Detected new or changed devices on smithi019 2024-01-24T14:21:18.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:18 smithi123 ceph-mon[44966]: Adjusting osd_memory_target on smithi019 to 17118M 2024-01-24T14:21:18.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:18 smithi123 ceph-mon[44966]: from='osd.0 [v2:172.21.15.19:6802/2456786579,v1:172.21.15.19:6803/2456786579]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]': finished 2024-01-24T14:21:18.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:18 smithi123 ceph-mon[44966]: osdmap e7: 1 total, 0 up, 1 in 2024-01-24T14:21:18.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:18 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-24T14:21:18.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:18 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-24T14:21:18.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:18 smithi123 ceph-mon[44966]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:21:18.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:18 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T14:21:18.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:18 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T14:21:18.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:18 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:21:18.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:18 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-24T14:21:18.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:18 smithi194 ceph-mon[45450]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:21:18.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:18 smithi194 ceph-mon[45450]: Detected new or changed devices on smithi019 2024-01-24T14:21:18.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:18 smithi194 ceph-mon[45450]: Adjusting osd_memory_target on smithi019 to 17118M 2024-01-24T14:21:18.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:18 smithi194 ceph-mon[45450]: from='osd.0 [v2:172.21.15.19:6802/2456786579,v1:172.21.15.19:6803/2456786579]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]': finished 2024-01-24T14:21:18.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:18 smithi194 ceph-mon[45450]: osdmap e7: 1 total, 0 up, 1 in 2024-01-24T14:21:18.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:18 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-24T14:21:18.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:18 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-24T14:21:18.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:18 smithi194 ceph-mon[45450]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:21:18.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:18 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T14:21:18.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:18 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T14:21:18.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:18 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:21:18.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:18 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-24T14:21:19.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:19 smithi019 ceph-mon[40663]: purged_snaps scrub starts 2024-01-24T14:21:19.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:19 smithi019 ceph-mon[40663]: purged_snaps scrub ok 2024-01-24T14:21:19.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:19 smithi019 ceph-mon[40663]: osd.0 [v2:172.21.15.19:6802/2456786579,v1:172.21.15.19:6803/2456786579] boot 2024-01-24T14:21:19.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:19 smithi019 ceph-mon[40663]: osdmap e8: 1 total, 1 up, 1 in 2024-01-24T14:21:19.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:19 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-24T14:21:19.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:19 smithi019 ceph-mon[40663]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:21:19.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:19 smithi123 ceph-mon[44966]: purged_snaps scrub starts 2024-01-24T14:21:19.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:19 smithi123 ceph-mon[44966]: purged_snaps scrub ok 2024-01-24T14:21:19.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:19 smithi123 ceph-mon[44966]: osd.0 [v2:172.21.15.19:6802/2456786579,v1:172.21.15.19:6803/2456786579] boot 2024-01-24T14:21:19.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:19 smithi123 ceph-mon[44966]: osdmap e8: 1 total, 1 up, 1 in 2024-01-24T14:21:19.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:19 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-24T14:21:19.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:19 smithi123 ceph-mon[44966]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:21:19.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:19 smithi194 ceph-mon[45450]: purged_snaps scrub starts 2024-01-24T14:21:19.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:19 smithi194 ceph-mon[45450]: purged_snaps scrub ok 2024-01-24T14:21:19.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:19 smithi194 ceph-mon[45450]: osd.0 [v2:172.21.15.19:6802/2456786579,v1:172.21.15.19:6803/2456786579] boot 2024-01-24T14:21:19.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:19 smithi194 ceph-mon[45450]: osdmap e8: 1 total, 1 up, 1 in 2024-01-24T14:21:19.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:19 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-01-24T14:21:19.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:19 smithi194 ceph-mon[45450]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-24T14:21:20.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:20 smithi019 ceph-mon[40663]: osdmap e9: 1 total, 1 up, 1 in 2024-01-24T14:21:20.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:20 smithi019 ceph-mon[40663]: from='client.? 172.21.15.123:0/2829804060' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "687f5036-a6d2-4aa1-86e7-7570e51ef179"}]: dispatch 2024-01-24T14:21:20.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:20 smithi019 ceph-mon[40663]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "687f5036-a6d2-4aa1-86e7-7570e51ef179"}]: dispatch 2024-01-24T14:21:20.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:20 smithi019 ceph-mon[40663]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "687f5036-a6d2-4aa1-86e7-7570e51ef179"}]': finished 2024-01-24T14:21:20.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:20 smithi019 ceph-mon[40663]: osdmap e10: 2 total, 1 up, 2 in 2024-01-24T14:21:20.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:20 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-24T14:21:20.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:20 smithi123 ceph-mon[44966]: osdmap e9: 1 total, 1 up, 1 in 2024-01-24T14:21:20.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:20 smithi123 ceph-mon[44966]: from='client.? 172.21.15.123:0/2829804060' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "687f5036-a6d2-4aa1-86e7-7570e51ef179"}]: dispatch 2024-01-24T14:21:20.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:20 smithi123 ceph-mon[44966]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "687f5036-a6d2-4aa1-86e7-7570e51ef179"}]: dispatch 2024-01-24T14:21:20.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:20 smithi123 ceph-mon[44966]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "687f5036-a6d2-4aa1-86e7-7570e51ef179"}]': finished 2024-01-24T14:21:20.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:20 smithi123 ceph-mon[44966]: osdmap e10: 2 total, 1 up, 2 in 2024-01-24T14:21:20.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:20 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-24T14:21:20.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:20 smithi194 ceph-mon[45450]: osdmap e9: 1 total, 1 up, 1 in 2024-01-24T14:21:20.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:20 smithi194 ceph-mon[45450]: from='client.? 172.21.15.123:0/2829804060' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "687f5036-a6d2-4aa1-86e7-7570e51ef179"}]: dispatch 2024-01-24T14:21:20.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:20 smithi194 ceph-mon[45450]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "687f5036-a6d2-4aa1-86e7-7570e51ef179"}]: dispatch 2024-01-24T14:21:20.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:20 smithi194 ceph-mon[45450]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "687f5036-a6d2-4aa1-86e7-7570e51ef179"}]': finished 2024-01-24T14:21:20.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:20 smithi194 ceph-mon[45450]: osdmap e10: 2 total, 1 up, 2 in 2024-01-24T14:21:20.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:20 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-24T14:21:21.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:21 smithi019 ceph-mon[40663]: from='client.? 172.21.15.123:0/2164592879' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T14:21:21.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:21 smithi019 ceph-mon[40663]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T14:21:21.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:21 smithi123 ceph-mon[44966]: from='client.? 172.21.15.123:0/2164592879' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T14:21:21.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:21 smithi123 ceph-mon[44966]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T14:21:21.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:21 smithi194 ceph-mon[45450]: from='client.? 172.21.15.123:0/2164592879' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T14:21:21.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:21 smithi194 ceph-mon[45450]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T14:21:24.235 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:23 smithi019 ceph-mon[40663]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T14:21:24.267 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:23 smithi123 ceph-mon[44966]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T14:21:24.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:23 smithi194 ceph-mon[45450]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T14:21:26.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:26 smithi019 ceph-mon[40663]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T14:21:26.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:26 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-01-24T14:21:26.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:26 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:21:26.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:26 smithi123 ceph-mon[44966]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T14:21:26.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:26 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-01-24T14:21:26.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:26 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:21:26.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:26 smithi194 ceph-mon[45450]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T14:21:26.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:26 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-01-24T14:21:26.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:26 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:21:27.530 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:27 smithi123 ceph-mon[44966]: Deploying daemon osd.1 on smithi123 2024-01-24T14:21:27.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:27 smithi019 ceph-mon[40663]: Deploying daemon osd.1 on smithi123 2024-01-24T14:21:27.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:27 smithi194 ceph-mon[45450]: Deploying daemon osd.1 on smithi123 2024-01-24T14:21:28.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:28 smithi123 ceph-mon[44966]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T14:21:28.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:28 smithi019 ceph-mon[40663]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T14:21:28.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:28 smithi194 ceph-mon[45450]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T14:21:30.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:30 smithi019 ceph-mon[40663]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T14:21:30.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:30 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:30.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:30 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:30.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:30 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:21:30.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:30 smithi123 ceph-mon[44966]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T14:21:30.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:30 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:30.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:30 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:30.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:30 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:21:30.843 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:30 smithi194 ceph-mon[45450]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T14:21:30.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:30 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:30.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:30 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:30.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:30 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:21:31.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:31.746+0000 7f82a6ffd700 1 -- 172.21.15.123:0/506867187 <== mgr.14152 v2:172.21.15.19:6800/1634350318 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f82b0061180 con 0x7f829c0a0510 2024-01-24T14:21:31.747 INFO:teuthology.orchestra.run.smithi123.stdout:Created osd(s) 1 on host 'smithi123' 2024-01-24T14:21:31.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:31.748+0000 7f82b80f8700 1 -- 172.21.15.123:0/506867187 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f829c0a0510 msgr2=0x7f829c0a29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:21:31.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:31.748+0000 7f82b80f8700 1 --2- 172.21.15.123:0/506867187 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f829c0a0510 0x7f829c0a29d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f82b019a290 tx=0x7f82a0009310 comp rx=0 tx=0).stop 2024-01-24T14:21:31.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:31.748+0000 7f82b80f8700 1 -- 172.21.15.123:0/506867187 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82b0104190 msgr2=0x7f82b01992f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:21:31.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:31.748+0000 7f82b80f8700 1 --2- 172.21.15.123:0/506867187 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82b0104190 0x7f82b01992f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f82ac00e9a0 tx=0x7f82ac00ecb0 comp rx=0 tx=0).stop 2024-01-24T14:21:31.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:31.748+0000 7f82b80f8700 1 -- 172.21.15.123:0/506867187 shutdown_connections 2024-01-24T14:21:31.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:31.748+0000 7f82b80f8700 1 --2- 172.21.15.123:0/506867187 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f829c0a0510 0x7f829c0a29d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:21:31.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:31.748+0000 7f82b80f8700 1 --2- 172.21.15.123:0/506867187 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f82b0106ac0 0x7f82b01a0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:21:31.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:31.748+0000 7f82b80f8700 1 --2- 172.21.15.123:0/506867187 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f82b0101860 0x7f82b0198db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:21:31.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:31.748+0000 7f82b80f8700 1 --2- 172.21.15.123:0/506867187 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82b0104190 0x7f82b01992f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:21:31.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:31.748+0000 7f82b80f8700 1 -- 172.21.15.123:0/506867187 >> 172.21.15.123:0/506867187 conn(0x7f82b00fb250 msgr2=0x7f82b0102590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:21:31.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:31.749+0000 7f82b80f8700 1 -- 172.21.15.123:0/506867187 shutdown_connections 2024-01-24T14:21:31.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:21:31.749+0000 7f82b80f8700 1 -- 172.21.15.123:0/506867187 wait complete. 2024-01-24T14:21:31.910 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:31 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:31.910 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:31 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:31.910 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:31 smithi123 ceph-mon[44966]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T14:21:31.910 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:31 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:21:31.910 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:31 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:21:31.910 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:31 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:31.910 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:31 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:31.910 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:31 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:31.910 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:31 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:21:31.985 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:31 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:31.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:31 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:31.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:31 smithi019 ceph-mon[40663]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T14:21:31.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:31 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:21:31.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:31 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:21:31.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:31 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:31.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:31 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:31.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:31 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:31.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:31 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:21:32.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:31 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:32.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:31 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:32.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:31 smithi194 ceph-mon[45450]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T14:21:32.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:31 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:21:32.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:31 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:21:32.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:31 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:32.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:31 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:32.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:31 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:32.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:31 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:21:32.117 DEBUG:teuthology.orchestra.run.smithi123:osd.1> sudo journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@osd.1.service 2024-01-24T14:21:32.119 INFO:tasks.cephadm:Deploying osd.2 on smithi194 with /dev/vg_nvme/lv_4... 2024-01-24T14:21:32.119 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-01-24T14:21:32.150 INFO:journalctl@ceph.osd.1.smithi123.stdout:-- Logs begin at Wed 2024-01-24 14:03:45 UTC. -- 2024-01-24T14:21:32.343 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/mon.c/config 2024-01-24T14:21:33.576 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-01-24T14:21:33.576 INFO:teuthology.orchestra.run.smithi194.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-01-24T14:21:33.576 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10+0 records in 2024-01-24T14:21:33.576 INFO:teuthology.orchestra.run.smithi194.stderr:10+0 records out 2024-01-24T14:21:33.576 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111796 s, 938 MB/s 2024-01-24T14:21:33.576 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping successful for: 2024-01-24T14:21:33.800 INFO:journalctl@ceph.osd.1.smithi123.stdout:Jan 24 14:21:33 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-1[49931]: 2024-01-24T14:21:33.423+0000 7fb69af94640 -1 osd.1 0 log_to_monitors true 2024-01-24T14:21:33.909 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 -- ceph orch daemon add osd smithi194:vg_nvme/lv_4 2024-01-24T14:21:34.127 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/mon.c/config 2024-01-24T14:21:34.129 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:33 smithi123 ceph-mon[44966]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T14:21:34.129 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:33 smithi123 ceph-mon[44966]: from='osd.1 [v2:172.21.15.123:6800/2407548889,v1:172.21.15.123:6801/2407548889]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-01-24T14:21:34.129 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:33 smithi123 ceph-mon[44966]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-01-24T14:21:34.228 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:33 smithi194 ceph-mon[45450]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T14:21:34.229 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:33 smithi194 ceph-mon[45450]: from='osd.1 [v2:172.21.15.123:6800/2407548889,v1:172.21.15.123:6801/2407548889]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-01-24T14:21:34.229 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:33 smithi194 ceph-mon[45450]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-01-24T14:21:34.235 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:33 smithi019 ceph-mon[40663]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T14:21:34.235 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:33 smithi019 ceph-mon[40663]: from='osd.1 [v2:172.21.15.123:6800/2407548889,v1:172.21.15.123:6801/2407548889]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-01-24T14:21:34.235 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:33 smithi019 ceph-mon[40663]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-01-24T14:21:35.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:35.057+0000 7f0f27a81700 1 -- 172.21.15.194:0/3554578825 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0f20102940 msgr2=0x7f0f20109260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:21:35.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:35.057+0000 7f0f27a81700 1 --2- 172.21.15.194:0/3554578825 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0f20102940 0x7f0f20109260 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0f1800b0a0 tx=0x7f0f1800b3b0 comp rx=0 tx=0).stop 2024-01-24T14:21:35.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:35.057+0000 7f0f27a81700 1 -- 172.21.15.194:0/3554578825 shutdown_connections 2024-01-24T14:21:35.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:35.057+0000 7f0f27a81700 1 --2- 172.21.15.194:0/3554578825 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0f20100e20 0x7f0f20101240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:21:35.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:35.057+0000 7f0f27a81700 1 --2- 172.21.15.194:0/3554578825 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0f20102940 0x7f0f20109260 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:21:35.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:35.057+0000 7f0f27a81700 1 --2- 172.21.15.194:0/3554578825 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f20101f80 0x7f0f20102400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:21:35.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:35.057+0000 7f0f27a81700 1 -- 172.21.15.194:0/3554578825 >> 172.21.15.194:0/3554578825 conn(0x7f0f200fc3c0 msgr2=0x7f0f200fe800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:21:35.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:35.057+0000 7f0f27a81700 1 -- 172.21.15.194:0/3554578825 shutdown_connections 2024-01-24T14:21:35.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:35.057+0000 7f0f27a81700 1 -- 172.21.15.194:0/3554578825 wait complete. 2024-01-24T14:21:35.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:35.058+0000 7f0f27a81700 1 Processor -- start 2024-01-24T14:21:35.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:35.058+0000 7f0f27a81700 1 -- start start 2024-01-24T14:21:35.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:35.058+0000 7f0f27a81700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0f20100e20 0x7f0f20196960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:21:35.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:35.058+0000 7f0f27a81700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f20101f80 0x7f0f20196ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:21:35.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:35.058+0000 7f0f27a81700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0f20102940 0x7f0f2019df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:21:35.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:35.058+0000 7f0f27a81700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2019e4f0 con 0x7f0f20100e20 2024-01-24T14:21:35.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:35.058+0000 7f0f27a81700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2019e630 con 0x7f0f20101f80 2024-01-24T14:21:35.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:35.058+0000 7f0f27a81700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2019e7c0 con 0x7f0f20102940 2024-01-24T14:21:35.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:35.058+0000 7f0f2601e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0f20102940 0x7f0f2019df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:21:35.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:35.058+0000 7f0f2601e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0f20102940 0x7f0f2019df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:50026/0 (socket says 172.21.15.194:50026) 2024-01-24T14:21:35.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:35.058+0000 7f0f2601e700 1 -- 172.21.15.194:0/4290001879 learned_addr learned my addr 172.21.15.194:0/4290001879 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-24T14:21:35.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:35.058+0000 7f0f2501c700 1 --2- 172.21.15.194:0/4290001879 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f20101f80 0x7f0f20196ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:21:35.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:35.058+0000 7f0f2581d700 1 --2- 172.21.15.194:0/4290001879 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0f20100e20 0x7f0f20196960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:21:35.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:35.058+0000 7f0f2601e700 1 -- 172.21.15.194:0/4290001879 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f20101f80 msgr2=0x7f0f20196ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:21:35.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:35.058+0000 7f0f2601e700 1 --2- 172.21.15.194:0/4290001879 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f20101f80 0x7f0f20196ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:21:35.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:35.058+0000 7f0f2601e700 1 -- 172.21.15.194:0/4290001879 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0f20100e20 msgr2=0x7f0f20196960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:21:35.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:35.058+0000 7f0f2601e700 1 --2- 172.21.15.194:0/4290001879 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0f20100e20 0x7f0f20196960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:21:35.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:35.058+0000 7f0f2601e700 1 -- 172.21.15.194:0/4290001879 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f18009d00 con 0x7f0f20102940 2024-01-24T14:21:35.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:35.059+0000 7f0f2601e700 1 --2- 172.21.15.194:0/4290001879 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0f20102940 0x7f0f2019df20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0f180126d0 tx=0x7f0f18012700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:21:35.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:35.059+0000 7f0f16ffd700 1 -- 172.21.15.194:0/4290001879 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f1800e040 con 0x7f0f20102940 2024-01-24T14:21:35.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:35.059+0000 7f0f16ffd700 1 -- 172.21.15.194:0/4290001879 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0f18012de0 con 0x7f0f20102940 2024-01-24T14:21:35.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:35.059+0000 7f0f27a81700 1 -- 172.21.15.194:0/4290001879 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f2019ea40 con 0x7f0f20102940 2024-01-24T14:21:35.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:35.059+0000 7f0f16ffd700 1 -- 172.21.15.194:0/4290001879 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f1801bb00 con 0x7f0f20102940 2024-01-24T14:21:35.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:35.059+0000 7f0f27a81700 1 -- 172.21.15.194:0/4290001879 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f2019ef00 con 0x7f0f20102940 2024-01-24T14:21:35.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:35.060+0000 7f0f27a81700 1 -- 172.21.15.194:0/4290001879 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f20108310 con 0x7f0f20102940 2024-01-24T14:21:35.064 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:35.061+0000 7f0f16ffd700 1 -- 172.21.15.194:0/4290001879 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 15) v1 ==== 135014+0+0 (secure 0 0 0) 0x7f0f1801bc80 con 0x7f0f20102940 2024-01-24T14:21:35.064 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:35.061+0000 7f0f16ffd700 1 --2- 172.21.15.194:0/4290001879 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f0f0c0a0510 0x7f0f0c0a29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:21:35.064 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:35.061+0000 7f0f16ffd700 1 -- 172.21.15.194:0/4290001879 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2029+0+0 (secure 0 0 0) 0x7f0f180cb510 con 0x7f0f20102940 2024-01-24T14:21:35.064 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:35.062+0000 7f0f2581d700 1 --2- 172.21.15.194:0/4290001879 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f0f0c0a0510 0x7f0f0c0a29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:21:35.064 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:35.062+0000 7f0f2581d700 1 --2- 172.21.15.194:0/4290001879 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f0f0c0a0510 0x7f0f0c0a29d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0f10005950 tx=0x7f0f1000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:21:35.067 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:35.066+0000 7f0f16ffd700 1 -- 172.21.15.194:0/4290001879 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f0f18096ab0 con 0x7f0f20102940 2024-01-24T14:21:35.261 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:35.260+0000 7f0f27a81700 1 -- 172.21.15.194:0/4290001879 --> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f0f20061180 con 0x7f0f0c0a0510 2024-01-24T14:21:35.299 INFO:journalctl@ceph.osd.1.smithi123.stdout:Jan 24 14:21:35 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-1[49931]: 2024-01-24T14:21:34.999+0000 7fb690dfa700 -1 osd.1 0 waiting for initial osdmap 2024-01-24T14:21:35.299 INFO:journalctl@ceph.osd.1.smithi123.stdout:Jan 24 14:21:35 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-1[49931]: 2024-01-24T14:21:35.003+0000 7fb68b3ec700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-24T14:21:35.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:34 smithi123 ceph-mon[44966]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-01-24T14:21:35.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:34 smithi123 ceph-mon[44966]: osdmap e11: 2 total, 1 up, 2 in 2024-01-24T14:21:35.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:34 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-24T14:21:35.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:34 smithi123 ceph-mon[44966]: from='osd.1 [v2:172.21.15.123:6800/2407548889,v1:172.21.15.123:6801/2407548889]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-01-24T14:21:35.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:34 smithi123 ceph-mon[44966]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-01-24T14:21:35.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:34 smithi123 ceph-mon[44966]: Detected new or changed devices on smithi123 2024-01-24T14:21:35.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:34 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:35.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:34 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:35.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:34 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:21:35.301 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:34 smithi123 ceph-mon[44966]: Adjusting osd_memory_target on smithi123 to 17114M 2024-01-24T14:21:35.301 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:34 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:35.301 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:34 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:21:35.301 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:34 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:21:35.301 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:34 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:35.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:34 smithi194 ceph-mon[45450]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-01-24T14:21:35.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:34 smithi194 ceph-mon[45450]: osdmap e11: 2 total, 1 up, 2 in 2024-01-24T14:21:35.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:34 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-24T14:21:35.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:34 smithi194 ceph-mon[45450]: from='osd.1 [v2:172.21.15.123:6800/2407548889,v1:172.21.15.123:6801/2407548889]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-01-24T14:21:35.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:34 smithi194 ceph-mon[45450]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-01-24T14:21:35.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:34 smithi194 ceph-mon[45450]: Detected new or changed devices on smithi123 2024-01-24T14:21:35.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:34 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:35.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:34 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:35.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:34 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:21:35.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:34 smithi194 ceph-mon[45450]: Adjusting osd_memory_target on smithi123 to 17114M 2024-01-24T14:21:35.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:34 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:35.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:34 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:21:35.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:34 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:21:35.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:34 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:35.487 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:34 smithi019 ceph-mon[40663]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-01-24T14:21:35.487 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:34 smithi019 ceph-mon[40663]: osdmap e11: 2 total, 1 up, 2 in 2024-01-24T14:21:35.487 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:34 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-24T14:21:35.487 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:34 smithi019 ceph-mon[40663]: from='osd.1 [v2:172.21.15.123:6800/2407548889,v1:172.21.15.123:6801/2407548889]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-01-24T14:21:35.488 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:34 smithi019 ceph-mon[40663]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-01-24T14:21:35.488 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:34 smithi019 ceph-mon[40663]: Detected new or changed devices on smithi123 2024-01-24T14:21:35.488 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:34 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:35.488 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:34 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:35.488 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:34 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:21:35.488 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:34 smithi019 ceph-mon[40663]: Adjusting osd_memory_target on smithi123 to 17114M 2024-01-24T14:21:35.488 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:34 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:35.488 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:34 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:21:35.488 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:34 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:21:35.488 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:34 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:36.262 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:36 smithi194 ceph-mon[45450]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T14:21:36.262 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:36 smithi194 ceph-mon[45450]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-01-24T14:21:36.262 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:36 smithi194 ceph-mon[45450]: osdmap e12: 2 total, 1 up, 2 in 2024-01-24T14:21:36.262 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:36 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-24T14:21:36.262 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:36 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-24T14:21:36.262 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:36 smithi194 ceph-mon[45450]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:21:36.262 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:36 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T14:21:36.262 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:36 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T14:21:36.262 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:36 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:21:36.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:36 smithi123 ceph-mon[44966]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T14:21:36.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:36 smithi123 ceph-mon[44966]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-01-24T14:21:36.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:36 smithi123 ceph-mon[44966]: osdmap e12: 2 total, 1 up, 2 in 2024-01-24T14:21:36.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:36 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-24T14:21:36.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:36 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-24T14:21:36.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:36 smithi123 ceph-mon[44966]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:21:36.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:36 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T14:21:36.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:36 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T14:21:36.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:36 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:21:36.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:36 smithi019 ceph-mon[40663]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T14:21:36.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:36 smithi019 ceph-mon[40663]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-01-24T14:21:36.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:36 smithi019 ceph-mon[40663]: osdmap e12: 2 total, 1 up, 2 in 2024-01-24T14:21:36.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:36 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-24T14:21:36.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:36 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-24T14:21:36.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:36 smithi019 ceph-mon[40663]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:21:36.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:36 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T14:21:36.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:36 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T14:21:36.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:36 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:21:37.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:37 smithi123 ceph-mon[44966]: purged_snaps scrub starts 2024-01-24T14:21:37.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:37 smithi123 ceph-mon[44966]: purged_snaps scrub ok 2024-01-24T14:21:37.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:37 smithi123 ceph-mon[44966]: osd.1 [v2:172.21.15.123:6800/2407548889,v1:172.21.15.123:6801/2407548889] boot 2024-01-24T14:21:37.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:37 smithi123 ceph-mon[44966]: osdmap e13: 2 total, 2 up, 2 in 2024-01-24T14:21:37.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:37 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-24T14:21:37.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:37 smithi123 ceph-mon[44966]: from='client.? 172.21.15.194:0/1504390630' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3e810904-371d-4ca8-b43c-434957a2aa12"}]: dispatch 2024-01-24T14:21:37.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:37 smithi123 ceph-mon[44966]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3e810904-371d-4ca8-b43c-434957a2aa12"}]: dispatch 2024-01-24T14:21:37.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:37 smithi123 ceph-mon[44966]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3e810904-371d-4ca8-b43c-434957a2aa12"}]': finished 2024-01-24T14:21:37.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:37 smithi123 ceph-mon[44966]: osdmap e14: 3 total, 2 up, 3 in 2024-01-24T14:21:37.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:37 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-24T14:21:37.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:37 smithi194 ceph-mon[45450]: purged_snaps scrub starts 2024-01-24T14:21:37.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:37 smithi194 ceph-mon[45450]: purged_snaps scrub ok 2024-01-24T14:21:37.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:37 smithi194 ceph-mon[45450]: osd.1 [v2:172.21.15.123:6800/2407548889,v1:172.21.15.123:6801/2407548889] boot 2024-01-24T14:21:37.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:37 smithi194 ceph-mon[45450]: osdmap e13: 2 total, 2 up, 2 in 2024-01-24T14:21:37.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:37 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-24T14:21:37.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:37 smithi194 ceph-mon[45450]: from='client.? 172.21.15.194:0/1504390630' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3e810904-371d-4ca8-b43c-434957a2aa12"}]: dispatch 2024-01-24T14:21:37.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:37 smithi194 ceph-mon[45450]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3e810904-371d-4ca8-b43c-434957a2aa12"}]: dispatch 2024-01-24T14:21:37.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:37 smithi194 ceph-mon[45450]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3e810904-371d-4ca8-b43c-434957a2aa12"}]': finished 2024-01-24T14:21:37.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:37 smithi194 ceph-mon[45450]: osdmap e14: 3 total, 2 up, 3 in 2024-01-24T14:21:37.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:37 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-24T14:21:37.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:37 smithi019 ceph-mon[40663]: purged_snaps scrub starts 2024-01-24T14:21:37.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:37 smithi019 ceph-mon[40663]: purged_snaps scrub ok 2024-01-24T14:21:37.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:37 smithi019 ceph-mon[40663]: osd.1 [v2:172.21.15.123:6800/2407548889,v1:172.21.15.123:6801/2407548889] boot 2024-01-24T14:21:37.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:37 smithi019 ceph-mon[40663]: osdmap e13: 2 total, 2 up, 2 in 2024-01-24T14:21:37.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:37 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-01-24T14:21:37.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:37 smithi019 ceph-mon[40663]: from='client.? 172.21.15.194:0/1504390630' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3e810904-371d-4ca8-b43c-434957a2aa12"}]: dispatch 2024-01-24T14:21:37.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:37 smithi019 ceph-mon[40663]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3e810904-371d-4ca8-b43c-434957a2aa12"}]: dispatch 2024-01-24T14:21:37.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:37 smithi019 ceph-mon[40663]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3e810904-371d-4ca8-b43c-434957a2aa12"}]': finished 2024-01-24T14:21:37.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:37 smithi019 ceph-mon[40663]: osdmap e14: 3 total, 2 up, 3 in 2024-01-24T14:21:37.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:37 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-24T14:21:38.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:38 smithi123 ceph-mon[44966]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T14:21:38.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:38 smithi123 ceph-mon[44966]: from='client.? 172.21.15.194:0/3672445640' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T14:21:38.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:38 smithi194 ceph-mon[45450]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T14:21:38.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:38 smithi194 ceph-mon[45450]: from='client.? 172.21.15.194:0/3672445640' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T14:21:38.485 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:38 smithi019 ceph-mon[40663]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-01-24T14:21:38.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:38 smithi019 ceph-mon[40663]: from='client.? 172.21.15.194:0/3672445640' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T14:21:40.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:40 smithi019 ceph-mon[40663]: pgmap v52: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T14:21:40.752 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:40 smithi194 ceph-mon[45450]: pgmap v52: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T14:21:40.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:40 smithi123 ceph-mon[44966]: pgmap v52: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T14:21:42.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:42 smithi194 ceph-mon[45450]: pgmap v53: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T14:21:42.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:42 smithi019 ceph-mon[40663]: pgmap v53: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T14:21:42.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:42 smithi123 ceph-mon[44966]: pgmap v53: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T14:21:43.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:43 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-01-24T14:21:43.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:43 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:21:43.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:43 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-01-24T14:21:43.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:43 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:21:43.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:43 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-01-24T14:21:43.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:43 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:21:44.648 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:44 smithi194 ceph-mon[45450]: Deploying daemon osd.2 on smithi194 2024-01-24T14:21:44.648 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:44 smithi194 ceph-mon[45450]: pgmap v54: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T14:21:44.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:44 smithi019 ceph-mon[40663]: Deploying daemon osd.2 on smithi194 2024-01-24T14:21:44.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:44 smithi019 ceph-mon[40663]: pgmap v54: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T14:21:44.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:44 smithi123 ceph-mon[44966]: Deploying daemon osd.2 on smithi194 2024-01-24T14:21:44.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:44 smithi123 ceph-mon[44966]: pgmap v54: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T14:21:46.672 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:46 smithi194 ceph-mon[45450]: pgmap v55: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T14:21:46.672 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:46 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:46.672 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:46 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:46.672 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:46 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:21:46.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:46 smithi019 ceph-mon[40663]: pgmap v55: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T14:21:46.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:46 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:46.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:46 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:46.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:46 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:21:46.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:46 smithi123 ceph-mon[44966]: pgmap v55: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T14:21:46.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:46 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:46.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:46 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:46.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:46 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:21:48.463 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:48.463+0000 7f0f16ffd700 1 -- 172.21.15.194:0/4290001879 <== mgr.14152 v2:172.21.15.19:6800/1634350318 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f0f20061180 con 0x7f0f0c0a0510 2024-01-24T14:21:48.464 INFO:teuthology.orchestra.run.smithi194.stdout:Created osd(s) 2 on host 'smithi194' 2024-01-24T14:21:48.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:48.465+0000 7f0f27a81700 1 -- 172.21.15.194:0/4290001879 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f0f0c0a0510 msgr2=0x7f0f0c0a29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:21:48.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:48.465+0000 7f0f27a81700 1 --2- 172.21.15.194:0/4290001879 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f0f0c0a0510 0x7f0f0c0a29d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0f10005950 tx=0x7f0f1000b410 comp rx=0 tx=0).stop 2024-01-24T14:21:48.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:48.465+0000 7f0f27a81700 1 -- 172.21.15.194:0/4290001879 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0f20102940 msgr2=0x7f0f2019df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:21:48.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:48.465+0000 7f0f27a81700 1 --2- 172.21.15.194:0/4290001879 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0f20102940 0x7f0f2019df20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0f180126d0 tx=0x7f0f18012700 comp rx=0 tx=0).stop 2024-01-24T14:21:48.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:48.465+0000 7f0f27a81700 1 -- 172.21.15.194:0/4290001879 shutdown_connections 2024-01-24T14:21:48.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:48.465+0000 7f0f27a81700 1 --2- 172.21.15.194:0/4290001879 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f0f0c0a0510 0x7f0f0c0a29d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:21:48.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:48.465+0000 7f0f27a81700 1 --2- 172.21.15.194:0/4290001879 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0f20100e20 0x7f0f20196960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:21:48.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:48.465+0000 7f0f27a81700 1 --2- 172.21.15.194:0/4290001879 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0f20102940 0x7f0f2019df20 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:21:48.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:48.465+0000 7f0f27a81700 1 --2- 172.21.15.194:0/4290001879 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f20101f80 0x7f0f20196ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:21:48.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:48.465+0000 7f0f27a81700 1 -- 172.21.15.194:0/4290001879 >> 172.21.15.194:0/4290001879 conn(0x7f0f200fc3c0 msgr2=0x7f0f20105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:21:48.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:48.465+0000 7f0f27a81700 1 -- 172.21.15.194:0/4290001879 shutdown_connections 2024-01-24T14:21:48.467 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:21:48.466+0000 7f0f27a81700 1 -- 172.21.15.194:0/4290001879 wait complete. 2024-01-24T14:21:48.659 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:48 smithi194 ceph-mon[45450]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T14:21:48.659 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:48 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:48.659 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:48 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:48.659 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:48 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:21:48.660 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:48 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:21:48.660 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:48 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:48.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:48 smithi019 ceph-mon[40663]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T14:21:48.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:48 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:48.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:48 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:48.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:48 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:21:48.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:48 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:21:48.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:48 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:48.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:48 smithi123 ceph-mon[44966]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T14:21:48.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:48 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:48.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:48 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:48.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:48 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:21:48.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:48 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:21:48.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:48 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:48.877 DEBUG:teuthology.orchestra.run.smithi194:osd.2> sudo journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@osd.2.service 2024-01-24T14:21:48.879 INFO:tasks.cephadm:Deploying osd.3 on smithi019 with /dev/vg_nvme/lv_3... 2024-01-24T14:21:48.879 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-01-24T14:21:48.910 INFO:journalctl@ceph.osd.2.smithi194.stdout:-- Logs begin at Wed 2024-01-24 14:04:01 UTC. -- 2024-01-24T14:21:49.136 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/mon.a/config 2024-01-24T14:21:49.551 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:49 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:49.551 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:49 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:49.552 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:49 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:21:49.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:49 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:49.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:49 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:49.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:49 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:21:49.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:49 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:49.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:49 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:49.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:49 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:21:50.429 INFO:teuthology.orchestra.run.smithi019.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-01-24T14:21:50.430 INFO:teuthology.orchestra.run.smithi019.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-01-24T14:21:50.430 INFO:teuthology.orchestra.run.smithi019.stderr: stderr: 10+0 records in 2024-01-24T14:21:50.430 INFO:teuthology.orchestra.run.smithi019.stderr:10+0 records out 2024-01-24T14:21:50.430 INFO:teuthology.orchestra.run.smithi019.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0139569 s, 751 MB/s 2024-01-24T14:21:50.430 INFO:teuthology.orchestra.run.smithi019.stderr:--> Zapping successful for: 2024-01-24T14:21:50.436 INFO:journalctl@ceph.osd.2.smithi194.stdout:Jan 24 14:21:50 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-2[51247]: 2024-01-24T14:21:50.157+0000 7f505add3640 -1 osd.2 0 log_to_monitors true 2024-01-24T14:21:50.692 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:50 smithi194 ceph-mon[45450]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T14:21:50.692 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:50 smithi194 ceph-mon[45450]: from='osd.2 [v2:172.21.15.194:6800/3632698637,v1:172.21.15.194:6801/3632698637]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-01-24T14:21:50.693 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:50 smithi194 ceph-mon[45450]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-01-24T14:21:50.714 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:50 smithi019 ceph-mon[40663]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T14:21:50.714 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:50 smithi019 ceph-mon[40663]: from='osd.2 [v2:172.21.15.194:6800/3632698637,v1:172.21.15.194:6801/3632698637]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-01-24T14:21:50.714 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:50 smithi019 ceph-mon[40663]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-01-24T14:21:50.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:50 smithi123 ceph-mon[44966]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T14:21:50.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:50 smithi123 ceph-mon[44966]: from='osd.2 [v2:172.21.15.194:6800/3632698637,v1:172.21.15.194:6801/3632698637]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-01-24T14:21:50.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:50 smithi123 ceph-mon[44966]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-01-24T14:21:50.840 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 -- ceph orch daemon add osd smithi019:vg_nvme/lv_3 2024-01-24T14:21:51.099 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/mon.a/config 2024-01-24T14:21:51.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:51 smithi019 ceph-mon[40663]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-01-24T14:21:51.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:51 smithi019 ceph-mon[40663]: osdmap e15: 3 total, 2 up, 3 in 2024-01-24T14:21:51.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:51 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-24T14:21:51.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:51 smithi019 ceph-mon[40663]: from='osd.2 [v2:172.21.15.194:6800/3632698637,v1:172.21.15.194:6801/3632698637]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-01-24T14:21:51.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:51 smithi019 ceph-mon[40663]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-01-24T14:21:51.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:51 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:51.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:51 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:51.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:51 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:21:51.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:51 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:51.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:51 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:21:51.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:51 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:21:51.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:51 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:51.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:51 smithi123 ceph-mon[44966]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-01-24T14:21:51.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:51 smithi123 ceph-mon[44966]: osdmap e15: 3 total, 2 up, 3 in 2024-01-24T14:21:51.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:51 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-24T14:21:51.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:51 smithi123 ceph-mon[44966]: from='osd.2 [v2:172.21.15.194:6800/3632698637,v1:172.21.15.194:6801/3632698637]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-01-24T14:21:51.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:51 smithi123 ceph-mon[44966]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-01-24T14:21:51.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:51 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:51.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:51 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:51.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:51 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:21:51.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:51 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:51.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:51 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:21:51.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:51 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:21:51.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:51 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:51.844 INFO:journalctl@ceph.osd.2.smithi194.stdout:Jan 24 14:21:51 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-2[51247]: 2024-01-24T14:21:51.457+0000 7f5050438700 -1 osd.2 0 waiting for initial osdmap 2024-01-24T14:21:51.844 INFO:journalctl@ceph.osd.2.smithi194.stdout:Jan 24 14:21:51 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-2[51247]: 2024-01-24T14:21:51.460+0000 7f5049a28700 -1 osd.2 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-24T14:21:51.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:51 smithi194 ceph-mon[45450]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-01-24T14:21:51.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:51 smithi194 ceph-mon[45450]: osdmap e15: 3 total, 2 up, 3 in 2024-01-24T14:21:51.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:51 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-24T14:21:51.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:51 smithi194 ceph-mon[45450]: from='osd.2 [v2:172.21.15.194:6800/3632698637,v1:172.21.15.194:6801/3632698637]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-01-24T14:21:51.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:51 smithi194 ceph-mon[45450]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-01-24T14:21:51.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:51 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:51.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:51 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:51.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:51 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:21:51.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:51 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:51.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:51 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:21:51.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:51 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:21:51.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:51 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:52.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:52.138+0000 7f66bb19a700 1 -- 172.21.15.19:0/2837796638 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f66b4101f80 msgr2=0x7f66b4102400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:21:52.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:52.138+0000 7f66bb19a700 1 --2- 172.21.15.19:0/2837796638 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f66b4101f80 0x7f66b4102400 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f66a0009990 tx=0x7f66a0009ca0 comp rx=0 tx=0).stop 2024-01-24T14:21:52.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:52.138+0000 7f66bb19a700 1 -- 172.21.15.19:0/2837796638 shutdown_connections 2024-01-24T14:21:52.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:52.138+0000 7f66bb19a700 1 --2- 172.21.15.19:0/2837796638 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f66b4102940 0x7f66b4109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:21:52.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:52.138+0000 7f66bb19a700 1 --2- 172.21.15.19:0/2837796638 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66b4100e20 0x7f66b4101240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:21:52.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:52.138+0000 7f66bb19a700 1 --2- 172.21.15.19:0/2837796638 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f66b4101f80 0x7f66b4102400 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:21:52.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:52.138+0000 7f66bb19a700 1 -- 172.21.15.19:0/2837796638 >> 172.21.15.19:0/2837796638 conn(0x7f66b40fc3c0 msgr2=0x7f66b40fe800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:21:52.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:52.138+0000 7f66bb19a700 1 -- 172.21.15.19:0/2837796638 shutdown_connections 2024-01-24T14:21:52.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:52.138+0000 7f66bb19a700 1 -- 172.21.15.19:0/2837796638 wait complete. 2024-01-24T14:21:52.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:52.139+0000 7f66bb19a700 1 Processor -- start 2024-01-24T14:21:52.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:52.139+0000 7f66bb19a700 1 -- start start 2024-01-24T14:21:52.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:52.139+0000 7f66bb19a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66b4100e20 0x7f66b4074ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:21:52.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:52.139+0000 7f66bb19a700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f66b4101f80 0x7f66b4073130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:21:52.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:52.139+0000 7f66bb19a700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f66b4102940 0x7f66b4073670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:21:52.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:52.139+0000 7f66bb19a700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b4073df0 con 0x7f66b4102940 2024-01-24T14:21:52.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:52.139+0000 7f66bb19a700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b4073f60 con 0x7f66b4100e20 2024-01-24T14:21:52.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:52.139+0000 7f66bb19a700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b41a0a80 con 0x7f66b4101f80 2024-01-24T14:21:52.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:52.139+0000 7f66b9737700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f66b4102940 0x7f66b4073670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:21:52.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:52.140+0000 7f66b9737700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f66b4102940 0x7f66b4073670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:57592/0 (socket says 172.21.15.19:57592) 2024-01-24T14:21:52.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:52.140+0000 7f66b9737700 1 -- 172.21.15.19:0/2097565734 learned_addr learned my addr 172.21.15.19:0/2097565734 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:21:52.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:52.140+0000 7f66b3fff700 1 --2- 172.21.15.19:0/2097565734 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f66b4101f80 0x7f66b4073130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:21:52.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:52.140+0000 7f66b8f36700 1 --2- 172.21.15.19:0/2097565734 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66b4100e20 0x7f66b4074ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:21:52.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:52.140+0000 7f66b9737700 1 -- 172.21.15.19:0/2097565734 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f66b4101f80 msgr2=0x7f66b4073130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:21:52.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:52.140+0000 7f66b9737700 1 --2- 172.21.15.19:0/2097565734 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f66b4101f80 0x7f66b4073130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:21:52.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:52.140+0000 7f66b9737700 1 -- 172.21.15.19:0/2097565734 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66b4100e20 msgr2=0x7f66b4074ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:21:52.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:52.140+0000 7f66b9737700 1 --2- 172.21.15.19:0/2097565734 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66b4100e20 0x7f66b4074ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:21:52.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:52.140+0000 7f66b9737700 1 -- 172.21.15.19:0/2097565734 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a00095f0 con 0x7f66b4102940 2024-01-24T14:21:52.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:52.140+0000 7f66b8f36700 1 --2- 172.21.15.19:0/2097565734 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66b4100e20 0x7f66b4074ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T14:21:52.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:52.140+0000 7f66b9737700 1 --2- 172.21.15.19:0/2097565734 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f66b4102940 0x7f66b4073670 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f66ac007ee0 tx=0x7f66ac00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:21:52.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:52.140+0000 7f66b1ffb700 1 -- 172.21.15.19:0/2097565734 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66ac017070 con 0x7f66b4102940 2024-01-24T14:21:52.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:52.140+0000 7f66b1ffb700 1 -- 172.21.15.19:0/2097565734 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f66ac00f040 con 0x7f66b4102940 2024-01-24T14:21:52.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:52.140+0000 7f66bb19a700 1 -- 172.21.15.19:0/2097565734 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66b41a0d60 con 0x7f66b4102940 2024-01-24T14:21:52.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:52.140+0000 7f66b1ffb700 1 -- 172.21.15.19:0/2097565734 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66ac005740 con 0x7f66b4102940 2024-01-24T14:21:52.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:52.141+0000 7f66bb19a700 1 -- 172.21.15.19:0/2097565734 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66b41a1200 con 0x7f66b4102940 2024-01-24T14:21:52.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:52.142+0000 7f66bb19a700 1 -- 172.21.15.19:0/2097565734 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66b4108360 con 0x7f66b4102940 2024-01-24T14:21:52.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:52.143+0000 7f66b1ffb700 1 -- 172.21.15.19:0/2097565734 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 15) v1 ==== 135014+0+0 (secure 0 0 0) 0x7f66ac00dc60 con 0x7f66b4102940 2024-01-24T14:21:52.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:52.144+0000 7f66b1ffb700 1 --2- 172.21.15.19:0/2097565734 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f66a40a0590 0x7f66a40a2a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:21:52.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:52.144+0000 7f66b8f36700 1 --2- 172.21.15.19:0/2097565734 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f66a40a0590 0x7f66a40a2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:21:52.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:52.144+0000 7f66b1ffb700 1 -- 172.21.15.19:0/2097565734 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2571+0+0 (secure 0 0 0) 0x7f66ac0ca070 con 0x7f66b4102940 2024-01-24T14:21:52.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:52.144+0000 7f66b8f36700 1 --2- 172.21.15.19:0/2097565734 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f66a40a0590 0x7f66a40a2a50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f66a8005fd0 tx=0x7f66a8005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:21:52.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:52.147+0000 7f66b1ffb700 1 -- 172.21.15.19:0/2097565734 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f66ac0952c0 con 0x7f66b4102940 2024-01-24T14:21:52.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:52.363+0000 7f66bb19a700 1 -- 172.21.15.19:0/2097565734 --> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f66b4061180 con 0x7f66a40a0590 2024-01-24T14:21:52.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:52 smithi019 ceph-mon[40663]: Detected new or changed devices on smithi194 2024-01-24T14:21:52.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:52 smithi019 ceph-mon[40663]: Adjusting osd_memory_target on smithi194 to 17114M 2024-01-24T14:21:52.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:52 smithi019 ceph-mon[40663]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T14:21:52.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:52 smithi019 ceph-mon[40663]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-01-24T14:21:52.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:52 smithi019 ceph-mon[40663]: osdmap e16: 3 total, 2 up, 3 in 2024-01-24T14:21:52.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:52 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-24T14:21:52.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:52 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-24T14:21:52.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:52 smithi019 ceph-mon[40663]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:21:52.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:52 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T14:21:52.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:52 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T14:21:52.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:52 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:21:52.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:52 smithi123 ceph-mon[44966]: Detected new or changed devices on smithi194 2024-01-24T14:21:52.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:52 smithi123 ceph-mon[44966]: Adjusting osd_memory_target on smithi194 to 17114M 2024-01-24T14:21:52.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:52 smithi123 ceph-mon[44966]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T14:21:52.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:52 smithi123 ceph-mon[44966]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-01-24T14:21:52.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:52 smithi123 ceph-mon[44966]: osdmap e16: 3 total, 2 up, 3 in 2024-01-24T14:21:52.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:52 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-24T14:21:52.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:52 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-24T14:21:52.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:52 smithi123 ceph-mon[44966]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:21:52.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:52 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T14:21:52.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:52 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T14:21:52.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:52 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:21:52.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:52 smithi194 ceph-mon[45450]: Detected new or changed devices on smithi194 2024-01-24T14:21:52.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:52 smithi194 ceph-mon[45450]: Adjusting osd_memory_target on smithi194 to 17114M 2024-01-24T14:21:52.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:52 smithi194 ceph-mon[45450]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T14:21:52.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:52 smithi194 ceph-mon[45450]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-01-24T14:21:52.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:52 smithi194 ceph-mon[45450]: osdmap e16: 3 total, 2 up, 3 in 2024-01-24T14:21:52.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:52 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-24T14:21:52.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:52 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-24T14:21:52.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:52 smithi194 ceph-mon[45450]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:21:52.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:52 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T14:21:52.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:52 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T14:21:52.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:52 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:21:53.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:53 smithi019 ceph-mon[40663]: purged_snaps scrub starts 2024-01-24T14:21:53.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:53 smithi019 ceph-mon[40663]: purged_snaps scrub ok 2024-01-24T14:21:53.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:53 smithi019 ceph-mon[40663]: osd.2 [v2:172.21.15.194:6800/3632698637,v1:172.21.15.194:6801/3632698637] boot 2024-01-24T14:21:53.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:53 smithi019 ceph-mon[40663]: osdmap e17: 3 total, 3 up, 3 in 2024-01-24T14:21:53.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:53 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-24T14:21:53.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:53 smithi019 ceph-mon[40663]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T14:21:53.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:53 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-01-24T14:21:53.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:53 smithi123 ceph-mon[44966]: purged_snaps scrub starts 2024-01-24T14:21:53.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:53 smithi123 ceph-mon[44966]: purged_snaps scrub ok 2024-01-24T14:21:53.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:53 smithi123 ceph-mon[44966]: osd.2 [v2:172.21.15.194:6800/3632698637,v1:172.21.15.194:6801/3632698637] boot 2024-01-24T14:21:53.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:53 smithi123 ceph-mon[44966]: osdmap e17: 3 total, 3 up, 3 in 2024-01-24T14:21:53.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:53 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-24T14:21:53.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:53 smithi123 ceph-mon[44966]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T14:21:53.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:53 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-01-24T14:21:53.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:53 smithi194 ceph-mon[45450]: purged_snaps scrub starts 2024-01-24T14:21:53.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:53 smithi194 ceph-mon[45450]: purged_snaps scrub ok 2024-01-24T14:21:53.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:53 smithi194 ceph-mon[45450]: osd.2 [v2:172.21.15.194:6800/3632698637,v1:172.21.15.194:6801/3632698637] boot 2024-01-24T14:21:53.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:53 smithi194 ceph-mon[45450]: osdmap e17: 3 total, 3 up, 3 in 2024-01-24T14:21:53.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:53 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-01-24T14:21:53.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:53 smithi194 ceph-mon[45450]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-01-24T14:21:53.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:53 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-01-24T14:21:54.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:54 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-01-24T14:21:54.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:54 smithi019 ceph-mon[40663]: osdmap e18: 3 total, 3 up, 3 in 2024-01-24T14:21:54.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:54 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-01-24T14:21:54.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:54 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/547370620' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3665d6e0-010a-40bd-b0c6-682af56a0cbb"}]: dispatch 2024-01-24T14:21:54.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:54 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-01-24T14:21:54.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:54 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/547370620' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3665d6e0-010a-40bd-b0c6-682af56a0cbb"}]': finished 2024-01-24T14:21:54.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:54 smithi019 ceph-mon[40663]: osdmap e19: 4 total, 3 up, 4 in 2024-01-24T14:21:54.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:54 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-24T14:21:54.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:54 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-01-24T14:21:54.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:54 smithi123 ceph-mon[44966]: osdmap e18: 3 total, 3 up, 3 in 2024-01-24T14:21:54.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:54 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-01-24T14:21:54.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:54 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/547370620' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3665d6e0-010a-40bd-b0c6-682af56a0cbb"}]: dispatch 2024-01-24T14:21:54.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:54 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-01-24T14:21:54.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:54 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/547370620' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3665d6e0-010a-40bd-b0c6-682af56a0cbb"}]': finished 2024-01-24T14:21:54.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:54 smithi123 ceph-mon[44966]: osdmap e19: 4 total, 3 up, 4 in 2024-01-24T14:21:54.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:54 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-24T14:21:54.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:54 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-01-24T14:21:54.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:54 smithi194 ceph-mon[45450]: osdmap e18: 3 total, 3 up, 3 in 2024-01-24T14:21:54.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:54 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-01-24T14:21:54.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:54 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/547370620' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3665d6e0-010a-40bd-b0c6-682af56a0cbb"}]: dispatch 2024-01-24T14:21:54.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:54 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-01-24T14:21:54.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:54 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/547370620' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3665d6e0-010a-40bd-b0c6-682af56a0cbb"}]': finished 2024-01-24T14:21:54.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:54 smithi194 ceph-mon[45450]: osdmap e19: 4 total, 3 up, 4 in 2024-01-24T14:21:54.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:54 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-24T14:21:55.235 INFO:journalctl@ceph.osd.0.smithi019.stdout:Jan 24 14:21:54 smithi019 sudo[65418]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-01-24T14:21:55.236 INFO:journalctl@ceph.osd.0.smithi019.stdout:Jan 24 14:21:54 smithi019 sudo[65418]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-24T14:21:55.236 INFO:journalctl@ceph.osd.0.smithi019.stdout:Jan 24 14:21:54 smithi019 sudo[65418]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-24T14:21:55.239 INFO:journalctl@ceph.osd.0.smithi019.stdout:Jan 24 14:21:54 smithi019 sudo[65418]: pam_unix(sudo:session): session closed for user root 2024-01-24T14:21:55.736 INFO:journalctl@ceph.osd.0.smithi019.stdout:Jan 24 14:21:55 smithi019 sudo[65469]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-01-24T14:21:55.736 INFO:journalctl@ceph.osd.0.smithi019.stdout:Jan 24 14:21:55 smithi019 sudo[65469]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-24T14:21:55.736 INFO:journalctl@ceph.osd.0.smithi019.stdout:Jan 24 14:21:55 smithi019 sudo[65469]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-24T14:21:55.736 INFO:journalctl@ceph.osd.0.smithi019.stdout:Jan 24 14:21:55 smithi019 sudo[65469]: pam_unix(sudo:session): session closed for user root 2024-01-24T14:21:55.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:55 smithi019 ceph-mon[40663]: pgmap v65: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-01-24T14:21:55.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:55 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/1719044170' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T14:21:55.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:55 smithi019 ceph-mon[40663]: osdmap e20: 4 total, 3 up, 4 in 2024-01-24T14:21:55.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:55 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-24T14:21:55.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:55 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:55.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:55 smithi123 ceph-mon[44966]: pgmap v65: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-01-24T14:21:55.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:55 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/1719044170' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T14:21:55.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:55 smithi123 ceph-mon[44966]: osdmap e20: 4 total, 3 up, 4 in 2024-01-24T14:21:55.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:55 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-24T14:21:55.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:55 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:55.800 INFO:journalctl@ceph.osd.1.smithi123.stdout:Jan 24 14:21:55 smithi123 sudo[55351]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-01-24T14:21:55.800 INFO:journalctl@ceph.osd.1.smithi123.stdout:Jan 24 14:21:55 smithi123 sudo[55351]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-24T14:21:55.800 INFO:journalctl@ceph.osd.1.smithi123.stdout:Jan 24 14:21:55 smithi123 sudo[55351]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-24T14:21:55.800 INFO:journalctl@ceph.osd.1.smithi123.stdout:Jan 24 14:21:55 smithi123 sudo[55351]: pam_unix(sudo:session): session closed for user root 2024-01-24T14:21:55.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:55 smithi194 ceph-mon[45450]: pgmap v65: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-01-24T14:21:55.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:55 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/1719044170' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T14:21:55.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:55 smithi194 ceph-mon[45450]: osdmap e20: 4 total, 3 up, 4 in 2024-01-24T14:21:55.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:55 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-24T14:21:55.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:55 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:56.299 INFO:journalctl@ceph.osd.1.smithi123.stdout:Jan 24 14:21:55 smithi123 sudo[55354]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-01-24T14:21:56.299 INFO:journalctl@ceph.osd.1.smithi123.stdout:Jan 24 14:21:55 smithi123 sudo[55354]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-24T14:21:56.299 INFO:journalctl@ceph.osd.1.smithi123.stdout:Jan 24 14:21:55 smithi123 sudo[55354]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-24T14:21:56.299 INFO:journalctl@ceph.osd.1.smithi123.stdout:Jan 24 14:21:55 smithi123 sudo[55354]: pam_unix(sudo:session): session closed for user root 2024-01-24T14:21:56.343 INFO:journalctl@ceph.osd.2.smithi194.stdout:Jan 24 14:21:56 smithi194 sudo[56670]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-01-24T14:21:56.344 INFO:journalctl@ceph.osd.2.smithi194.stdout:Jan 24 14:21:56 smithi194 sudo[56670]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-24T14:21:56.344 INFO:journalctl@ceph.osd.2.smithi194.stdout:Jan 24 14:21:56 smithi194 sudo[56670]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-24T14:21:56.344 INFO:journalctl@ceph.osd.2.smithi194.stdout:Jan 24 14:21:56 smithi194 sudo[56670]: pam_unix(sudo:session): session closed for user root 2024-01-24T14:21:56.344 INFO:journalctl@ceph.osd.2.smithi194.stdout:Jan 24 14:21:56 smithi194 sudo[56673]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-01-24T14:21:56.344 INFO:journalctl@ceph.osd.2.smithi194.stdout:Jan 24 14:21:56 smithi194 sudo[56673]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-24T14:21:56.344 INFO:journalctl@ceph.osd.2.smithi194.stdout:Jan 24 14:21:56 smithi194 sudo[56673]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-24T14:21:56.344 INFO:journalctl@ceph.osd.2.smithi194.stdout:Jan 24 14:21:56 smithi194 sudo[56673]: pam_unix(sudo:session): session closed for user root 2024-01-24T14:21:57.235 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:56 smithi019 sudo[66213]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-01-24T14:21:57.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:56 smithi019 sudo[66213]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-24T14:21:57.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:56 smithi019 sudo[66213]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-24T14:21:57.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:56 smithi019 sudo[66213]: pam_unix(sudo:session): session closed for user root 2024-01-24T14:21:57.294 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:21:57.294+0000 7f66b1ffb700 1 -- 172.21.15.19:0/2097565734 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f66ac095690 con 0x7f66b4102940 2024-01-24T14:21:57.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:57 smithi019 sudo[66403]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-01-24T14:21:57.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:57 smithi019 sudo[66403]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-24T14:21:57.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:57 smithi019 sudo[66403]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-24T14:21:57.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:57 smithi019 sudo[66403]: pam_unix(sudo:session): session closed for user root 2024-01-24T14:21:57.775 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:57 smithi194 sudo[56676]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-01-24T14:21:57.775 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:57 smithi194 sudo[56676]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-24T14:21:57.775 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:57 smithi194 sudo[56676]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-24T14:21:57.775 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:57 smithi194 sudo[56676]: pam_unix(sudo:session): session closed for user root 2024-01-24T14:21:57.775 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:57 smithi194 sudo[56679]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-01-24T14:21:57.775 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:57 smithi194 sudo[56679]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-24T14:21:57.775 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:57 smithi194 sudo[56679]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-24T14:21:57.775 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:57 smithi194 sudo[56679]: pam_unix(sudo:session): session closed for user root 2024-01-24T14:21:58.049 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:57 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:58.049 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:57 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:58.049 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:57 smithi123 ceph-mon[44966]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-01-24T14:21:58.049 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:58 smithi123 sudo[55357]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-01-24T14:21:58.050 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:58 smithi123 sudo[55357]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-24T14:21:58.050 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:58 smithi123 sudo[55357]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-24T14:21:58.050 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:58 smithi123 sudo[55357]: pam_unix(sudo:session): session closed for user root 2024-01-24T14:21:58.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:57 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:58.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:57 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:58.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:57 smithi194 ceph-mon[45450]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-01-24T14:21:58.142 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:57 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:58.142 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:57 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:21:58.142 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:57 smithi019 ceph-mon[40663]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-01-24T14:21:58.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:58 smithi123 sudo[55360]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-01-24T14:21:58.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:58 smithi123 sudo[55360]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-01-24T14:21:58.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:58 smithi123 sudo[55360]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-01-24T14:21:58.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:58 smithi123 sudo[55360]: pam_unix(sudo:session): session closed for user root 2024-01-24T14:21:59.049 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:58 smithi123 ceph-mon[44966]: pgmap v67: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-01-24T14:21:59.049 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:58 smithi123 ceph-mon[44966]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-01-24T14:21:59.050 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:58 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-01-24T14:21:59.050 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:58 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T14:21:59.050 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:58 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T14:21:59.050 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:58 smithi123 ceph-mon[44966]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-01-24T14:21:59.050 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:58 smithi123 ceph-mon[44966]: mgrmap e16: x(active, since 114s), standbys: z, y 2024-01-24T14:21:59.050 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:58 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-01-24T14:21:59.050 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:58 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T14:21:59.050 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:58 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T14:21:59.050 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:58 smithi123 ceph-mon[44966]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-01-24T14:21:59.050 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:58 smithi123 ceph-mon[44966]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-01-24T14:21:59.050 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:58 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-01-24T14:21:59.050 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:58 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T14:21:59.050 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:58 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T14:21:59.051 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:21:58 smithi123 ceph-mon[44966]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-01-24T14:21:59.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:58 smithi194 ceph-mon[45450]: pgmap v67: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-01-24T14:21:59.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:58 smithi194 ceph-mon[45450]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-01-24T14:21:59.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:58 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-01-24T14:21:59.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:58 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T14:21:59.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:58 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T14:21:59.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:58 smithi194 ceph-mon[45450]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-01-24T14:21:59.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:58 smithi194 ceph-mon[45450]: mgrmap e16: x(active, since 114s), standbys: z, y 2024-01-24T14:21:59.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:58 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-01-24T14:21:59.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:58 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T14:21:59.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:58 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T14:21:59.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:58 smithi194 ceph-mon[45450]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-01-24T14:21:59.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:58 smithi194 ceph-mon[45450]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-01-24T14:21:59.095 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:58 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-01-24T14:21:59.095 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:58 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T14:21:59.095 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:58 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T14:21:59.095 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:21:58 smithi194 ceph-mon[45450]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-01-24T14:21:59.109 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:58 smithi019 ceph-mon[40663]: pgmap v67: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-01-24T14:21:59.109 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:58 smithi019 ceph-mon[40663]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-01-24T14:21:59.109 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:58 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-01-24T14:21:59.109 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:58 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T14:21:59.109 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:58 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T14:21:59.109 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:58 smithi019 ceph-mon[40663]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-01-24T14:21:59.109 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:58 smithi019 ceph-mon[40663]: mgrmap e16: x(active, since 114s), standbys: z, y 2024-01-24T14:21:59.110 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:58 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-01-24T14:21:59.110 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:58 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T14:21:59.110 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:58 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T14:21:59.110 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:58 smithi019 ceph-mon[40663]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-01-24T14:21:59.110 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:58 smithi019 ceph-mon[40663]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-01-24T14:21:59.110 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:58 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-01-24T14:21:59.110 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:58 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-01-24T14:21:59.110 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:58 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-01-24T14:21:59.110 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:21:58 smithi019 ceph-mon[40663]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-01-24T14:22:00.485 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:00 smithi019 ceph-mon[40663]: pgmap v68: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-01-24T14:22:00.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:00 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-01-24T14:22:00.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:00 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:22:00.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:00 smithi123 ceph-mon[44966]: pgmap v68: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-01-24T14:22:00.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:00 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-01-24T14:22:00.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:00 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:22:00.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:00 smithi194 ceph-mon[45450]: pgmap v68: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-01-24T14:22:00.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:00 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-01-24T14:22:00.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:00 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:22:01.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:01 smithi019 ceph-mon[40663]: Deploying daemon osd.3 on smithi019 2024-01-24T14:22:01.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:01 smithi123 ceph-mon[44966]: Deploying daemon osd.3 on smithi019 2024-01-24T14:22:01.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:01 smithi194 ceph-mon[45450]: Deploying daemon osd.3 on smithi019 2024-01-24T14:22:02.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:02 smithi019 ceph-mon[40663]: pgmap v69: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-01-24T14:22:02.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:02 smithi123 ceph-mon[44966]: pgmap v69: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-01-24T14:22:02.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:02 smithi194 ceph-mon[45450]: pgmap v69: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-01-24T14:22:04.671 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:04 smithi019 ceph-mon[40663]: pgmap v70: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 65 KiB/s wr, 5 op/s 2024-01-24T14:22:04.671 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:04 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:04.671 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:04 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:04.671 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:04 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:22:04.802 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:04 smithi123 ceph-mon[44966]: pgmap v70: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 65 KiB/s wr, 5 op/s 2024-01-24T14:22:04.802 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:04 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:04.802 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:04 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:04.802 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:04 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:22:04.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:04 smithi194 ceph-mon[45450]: pgmap v70: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 65 KiB/s wr, 5 op/s 2024-01-24T14:22:04.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:04 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:04.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:04 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:04.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:04 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:22:06.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:06.457+0000 7f66b1ffb700 1 -- 172.21.15.19:0/2097565734 <== mgr.14152 v2:172.21.15.19:6800/1634350318 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f66b4061180 con 0x7f66a40a0590 2024-01-24T14:22:06.459 INFO:teuthology.orchestra.run.smithi019.stdout:Created osd(s) 3 on host 'smithi019' 2024-01-24T14:22:06.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:06.460+0000 7f66bb19a700 1 -- 172.21.15.19:0/2097565734 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f66a40a0590 msgr2=0x7f66a40a2a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:22:06.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:06.460+0000 7f66bb19a700 1 --2- 172.21.15.19:0/2097565734 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f66a40a0590 0x7f66a40a2a50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f66a8005fd0 tx=0x7f66a8005e30 comp rx=0 tx=0).stop 2024-01-24T14:22:06.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:06.460+0000 7f66bb19a700 1 -- 172.21.15.19:0/2097565734 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f66b4102940 msgr2=0x7f66b4073670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:22:06.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:06.460+0000 7f66bb19a700 1 --2- 172.21.15.19:0/2097565734 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f66b4102940 0x7f66b4073670 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f66ac007ee0 tx=0x7f66ac00d3b0 comp rx=0 tx=0).stop 2024-01-24T14:22:06.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:06.460+0000 7f66bb19a700 1 -- 172.21.15.19:0/2097565734 shutdown_connections 2024-01-24T14:22:06.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:06.460+0000 7f66bb19a700 1 --2- 172.21.15.19:0/2097565734 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f66a40a0590 0x7f66a40a2a50 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:22:06.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:06.460+0000 7f66bb19a700 1 --2- 172.21.15.19:0/2097565734 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f66b4102940 0x7f66b4073670 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:22:06.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:06.460+0000 7f66bb19a700 1 --2- 172.21.15.19:0/2097565734 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66b4100e20 0x7f66b4074ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:22:06.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:06.460+0000 7f66bb19a700 1 --2- 172.21.15.19:0/2097565734 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f66b4101f80 0x7f66b4073130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:22:06.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:06.460+0000 7f66bb19a700 1 -- 172.21.15.19:0/2097565734 >> 172.21.15.19:0/2097565734 conn(0x7f66b40fc3c0 msgr2=0x7f66b41054a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:22:06.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:06.460+0000 7f66bb19a700 1 -- 172.21.15.19:0/2097565734 shutdown_connections 2024-01-24T14:22:06.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:06.460+0000 7f66bb19a700 1 -- 172.21.15.19:0/2097565734 wait complete. 2024-01-24T14:22:06.624 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:06 smithi019 ceph-mon[40663]: pgmap v71: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-01-24T14:22:06.624 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:06 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:06.624 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:06 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:06.624 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:06 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:22:06.625 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:06 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:22:06.625 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:06 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:06.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:06 smithi123 ceph-mon[44966]: pgmap v71: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-01-24T14:22:06.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:06 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:06.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:06 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:06.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:06 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:22:06.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:06 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:22:06.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:06 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:06.844 DEBUG:teuthology.orchestra.run.smithi019:osd.3> sudo journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@osd.3.service 2024-01-24T14:22:06.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:06 smithi194 ceph-mon[45450]: pgmap v71: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-01-24T14:22:06.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:06 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:06.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:06 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:06.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:06 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:22:06.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:06 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:22:06.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:06 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:06.847 INFO:tasks.cephadm:Deploying osd.4 on smithi123 with /dev/vg_nvme/lv_3... 2024-01-24T14:22:06.847 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-01-24T14:22:06.884 INFO:journalctl@ceph.osd.3.smithi019.stdout:-- Logs begin at Wed 2024-01-24 14:03:52 UTC. -- 2024-01-24T14:22:07.088 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/mon.b/config 2024-01-24T14:22:07.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:07 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:07.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:07 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:07.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:07 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:22:07.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:07 smithi123 ceph-mon[44966]: pgmap v72: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-01-24T14:22:07.720 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:07 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:07.720 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:07 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:07.720 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:07 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:22:07.721 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:07 smithi019 ceph-mon[40663]: pgmap v72: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-01-24T14:22:07.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:07 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:07.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:07 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:07.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:07 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:22:07.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:07 smithi194 ceph-mon[45450]: pgmap v72: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-01-24T14:22:08.384 INFO:teuthology.orchestra.run.smithi123.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-01-24T14:22:08.384 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-01-24T14:22:08.384 INFO:teuthology.orchestra.run.smithi123.stderr: stderr: 10+0 records in 2024-01-24T14:22:08.384 INFO:teuthology.orchestra.run.smithi123.stderr:10+0 records out 2024-01-24T14:22:08.384 INFO:teuthology.orchestra.run.smithi123.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124935 s, 839 MB/s 2024-01-24T14:22:08.385 INFO:teuthology.orchestra.run.smithi123.stderr:--> Zapping successful for: 2024-01-24T14:22:08.459 INFO:journalctl@ceph.osd.3.smithi019.stdout:Jan 24 14:22:08 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-3[67628]: 2024-01-24T14:22:08.162+0000 7fb0cba0c640 -1 osd.3 0 log_to_monitors true 2024-01-24T14:22:08.690 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:08 smithi123 ceph-mon[44966]: from='osd.3 [v2:172.21.15.19:6810/3868219159,v1:172.21.15.19:6811/3868219159]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-01-24T14:22:08.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:08 smithi019 ceph-mon[40663]: from='osd.3 [v2:172.21.15.19:6810/3868219159,v1:172.21.15.19:6811/3868219159]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-01-24T14:22:08.758 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 -- ceph orch daemon add osd smithi123:vg_nvme/lv_3 2024-01-24T14:22:08.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:08 smithi194 ceph-mon[45450]: from='osd.3 [v2:172.21.15.19:6810/3868219159,v1:172.21.15.19:6811/3868219159]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-01-24T14:22:08.973 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/mon.b/config 2024-01-24T14:22:09.736 INFO:journalctl@ceph.osd.3.smithi019.stdout:Jan 24 14:22:09 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-3[67628]: 2024-01-24T14:22:09.478+0000 7fb0c1872700 -1 osd.3 0 waiting for initial osdmap 2024-01-24T14:22:09.736 INFO:journalctl@ceph.osd.3.smithi019.stdout:Jan 24 14:22:09 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-3[67628]: 2024-01-24T14:22:09.485+0000 7fb0b965f700 -1 osd.3 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-24T14:22:09.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:09 smithi019 ceph-mon[40663]: from='osd.3 [v2:172.21.15.19:6810/3868219159,v1:172.21.15.19:6811/3868219159]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-01-24T14:22:09.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:09 smithi019 ceph-mon[40663]: osdmap e21: 4 total, 3 up, 4 in 2024-01-24T14:22:09.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:09 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-24T14:22:09.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:09 smithi019 ceph-mon[40663]: from='osd.3 [v2:172.21.15.19:6810/3868219159,v1:172.21.15.19:6811/3868219159]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-01-24T14:22:09.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:09 smithi019 ceph-mon[40663]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-01-24T14:22:09.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:09 smithi019 ceph-mon[40663]: Detected new or changed devices on smithi019 2024-01-24T14:22:09.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:09 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:09.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:09 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:09.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:09 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:22:09.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:09 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:22:09.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:09 smithi019 ceph-mon[40663]: Adjusting osd_memory_target on smithi019 to 8559M 2024-01-24T14:22:09.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:09 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:09.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:09 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:22:09.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:09 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:22:09.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:09 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:09.746 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:09 smithi123 ceph-mon[44966]: from='osd.3 [v2:172.21.15.19:6810/3868219159,v1:172.21.15.19:6811/3868219159]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-01-24T14:22:09.746 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:09 smithi123 ceph-mon[44966]: osdmap e21: 4 total, 3 up, 4 in 2024-01-24T14:22:09.746 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:09 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-24T14:22:09.746 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:09 smithi123 ceph-mon[44966]: from='osd.3 [v2:172.21.15.19:6810/3868219159,v1:172.21.15.19:6811/3868219159]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-01-24T14:22:09.746 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:09 smithi123 ceph-mon[44966]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-01-24T14:22:09.746 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:09 smithi123 ceph-mon[44966]: Detected new or changed devices on smithi019 2024-01-24T14:22:09.747 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:09 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:09.747 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:09 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:09.747 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:09 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:22:09.747 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:09 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:22:09.747 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:09 smithi123 ceph-mon[44966]: Adjusting osd_memory_target on smithi019 to 8559M 2024-01-24T14:22:09.747 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:09 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:09.747 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:09 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:22:09.747 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:09 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:22:09.747 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:09 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:09.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:09 smithi194 ceph-mon[45450]: from='osd.3 [v2:172.21.15.19:6810/3868219159,v1:172.21.15.19:6811/3868219159]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-01-24T14:22:09.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:09 smithi194 ceph-mon[45450]: osdmap e21: 4 total, 3 up, 4 in 2024-01-24T14:22:09.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:09 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-24T14:22:09.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:09 smithi194 ceph-mon[45450]: from='osd.3 [v2:172.21.15.19:6810/3868219159,v1:172.21.15.19:6811/3868219159]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-01-24T14:22:09.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:09 smithi194 ceph-mon[45450]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-01-24T14:22:09.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:09 smithi194 ceph-mon[45450]: Detected new or changed devices on smithi019 2024-01-24T14:22:09.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:09 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:09.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:09 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:09.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:09 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:22:09.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:09 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:22:09.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:09 smithi194 ceph-mon[45450]: Adjusting osd_memory_target on smithi019 to 8559M 2024-01-24T14:22:09.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:09 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:09.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:09 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:22:09.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:09 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:22:09.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:09 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:09.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:09.922+0000 7f1bb3207700 1 -- 172.21.15.123:0/639490667 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1bac106b60 msgr2=0x7f1bac108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:22:09.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:09.922+0000 7f1bb3207700 1 --2- 172.21.15.123:0/639490667 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1bac106b60 0x7f1bac108f50 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1ba400b0a0 tx=0x7f1ba400b3b0 comp rx=0 tx=0).stop 2024-01-24T14:22:09.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:09.922+0000 7f1bb3207700 1 -- 172.21.15.123:0/639490667 shutdown_connections 2024-01-24T14:22:09.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:09.922+0000 7f1bb3207700 1 --2- 172.21.15.123:0/639490667 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1bac101900 0x7f1bac103cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:22:09.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:09.922+0000 7f1bb3207700 1 --2- 172.21.15.123:0/639490667 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1bac106b60 0x7f1bac108f50 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:22:09.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:09.922+0000 7f1bb3207700 1 --2- 172.21.15.123:0/639490667 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1bac104230 0x7f1bac106620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:22:09.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:09.922+0000 7f1bb3207700 1 -- 172.21.15.123:0/639490667 >> 172.21.15.123:0/639490667 conn(0x7f1bac0fb250 msgr2=0x7f1bac0fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:22:09.923 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:09.922+0000 7f1bb3207700 1 -- 172.21.15.123:0/639490667 shutdown_connections 2024-01-24T14:22:09.924 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:09.922+0000 7f1bb3207700 1 -- 172.21.15.123:0/639490667 wait complete. 2024-01-24T14:22:09.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:09.923+0000 7f1bb3207700 1 Processor -- start 2024-01-24T14:22:09.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:09.923+0000 7f1bb3207700 1 -- start start 2024-01-24T14:22:09.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:09.923+0000 7f1bb3207700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1bac101900 0x7f1bac196be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:22:09.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:09.923+0000 7f1bb3207700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1bac104230 0x7f1bac197120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:22:09.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:09.923+0000 7f1bb3207700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1bac106b60 0x7f1bac19e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:22:09.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:09.923+0000 7f1bb3207700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bac19e770 con 0x7f1bac106b60 2024-01-24T14:22:09.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:09.923+0000 7f1bb3207700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bac19e8b0 con 0x7f1bac104230 2024-01-24T14:22:09.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:09.923+0000 7f1bb3207700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bac19ea40 con 0x7f1bac101900 2024-01-24T14:22:09.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:09.923+0000 7f1babfff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1bac104230 0x7f1bac197120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:22:09.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:09.923+0000 7f1babfff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1bac104230 0x7f1bac197120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:54480/0 (socket says 172.21.15.123:54480) 2024-01-24T14:22:09.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:09.923+0000 7f1babfff700 1 -- 172.21.15.123:0/4226335993 learned_addr learned my addr 172.21.15.123:0/4226335993 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-24T14:22:09.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:09.923+0000 7f1bb17a4700 1 --2- 172.21.15.123:0/4226335993 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1bac106b60 0x7f1bac19e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:22:09.926 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:09.923+0000 7f1bb0fa3700 1 --2- 172.21.15.123:0/4226335993 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1bac101900 0x7f1bac196be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:22:09.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:09.923+0000 7f1babfff700 1 -- 172.21.15.123:0/4226335993 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1bac101900 msgr2=0x7f1bac196be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:22:09.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:09.923+0000 7f1babfff700 1 --2- 172.21.15.123:0/4226335993 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1bac101900 0x7f1bac196be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:22:09.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:09.923+0000 7f1babfff700 1 -- 172.21.15.123:0/4226335993 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1bac106b60 msgr2=0x7f1bac19e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:22:09.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:09.923+0000 7f1babfff700 1 --2- 172.21.15.123:0/4226335993 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1bac106b60 0x7f1bac19e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:22:09.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:09.923+0000 7f1babfff700 1 -- 172.21.15.123:0/4226335993 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba4009d00 con 0x7f1bac104230 2024-01-24T14:22:09.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:09.923+0000 7f1bb0fa3700 1 --2- 172.21.15.123:0/4226335993 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1bac101900 0x7f1bac196be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T14:22:09.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:09.924+0000 7f1babfff700 1 --2- 172.21.15.123:0/4226335993 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1bac104230 0x7f1bac197120 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1b9800b530 tx=0x7f1b9800b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:22:09.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:09.924+0000 7f1ba9ffb700 1 -- 172.21.15.123:0/4226335993 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b98011830 con 0x7f1bac104230 2024-01-24T14:22:09.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:09.924+0000 7f1bb3207700 1 -- 172.21.15.123:0/4226335993 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bac19ed20 con 0x7f1bac104230 2024-01-24T14:22:09.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:09.924+0000 7f1ba9ffb700 1 -- 172.21.15.123:0/4226335993 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1b98011e50 con 0x7f1bac104230 2024-01-24T14:22:09.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:09.924+0000 7f1ba9ffb700 1 -- 172.21.15.123:0/4226335993 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b9800f2e0 con 0x7f1bac104230 2024-01-24T14:22:09.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:09.924+0000 7f1bb3207700 1 -- 172.21.15.123:0/4226335993 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bac19f1c0 con 0x7f1bac104230 2024-01-24T14:22:09.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:09.925+0000 7f1bb3207700 1 -- 172.21.15.123:0/4226335993 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bac0fcdd0 con 0x7f1bac104230 2024-01-24T14:22:09.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:09.926+0000 7f1ba9ffb700 1 -- 172.21.15.123:0/4226335993 <== mon.2 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f1b98010380 con 0x7f1bac104230 2024-01-24T14:22:09.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:09.926+0000 7f1ba9ffb700 1 --2- 172.21.15.123:0/4226335993 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f1b9c0a05a0 0x7f1b9c0a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:22:09.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:09.926+0000 7f1ba9ffb700 1 -- 172.21.15.123:0/4226335993 <== mon.2 v2:172.21.15.123:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 3389+0+0 (secure 0 0 0) 0x7f1b980ca230 con 0x7f1bac104230 2024-01-24T14:22:09.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:09.927+0000 7f1bb0fa3700 1 --2- 172.21.15.123:0/4226335993 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f1b9c0a05a0 0x7f1b9c0a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:22:09.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:09.927+0000 7f1bb0fa3700 1 --2- 172.21.15.123:0/4226335993 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f1b9c0a05a0 0x7f1b9c0a2a60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1ba0005d90 tx=0x7f1ba0005d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:22:09.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:09.931+0000 7f1ba9ffb700 1 -- 172.21.15.123:0/4226335993 <== mon.2 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f1b98095070 con 0x7f1bac104230 2024-01-24T14:22:10.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:10.126+0000 7f1bb3207700 1 -- 172.21.15.123:0/4226335993 --> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f1bac061180 con 0x7f1b9c0a05a0 2024-01-24T14:22:10.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:10 smithi019 ceph-mon[40663]: from='osd.3 [v2:172.21.15.19:6810/3868219159,v1:172.21.15.19:6811/3868219159]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]': finished 2024-01-24T14:22:10.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:10 smithi019 ceph-mon[40663]: osdmap e22: 4 total, 3 up, 4 in 2024-01-24T14:22:10.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:10 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-24T14:22:10.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:10 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-24T14:22:10.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:10 smithi019 ceph-mon[40663]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:22:10.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:10 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T14:22:10.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:10 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T14:22:10.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:10 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:22:10.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:10 smithi123 ceph-mon[44966]: from='osd.3 [v2:172.21.15.19:6810/3868219159,v1:172.21.15.19:6811/3868219159]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]': finished 2024-01-24T14:22:10.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:10 smithi123 ceph-mon[44966]: osdmap e22: 4 total, 3 up, 4 in 2024-01-24T14:22:10.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:10 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-24T14:22:10.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:10 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-24T14:22:10.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:10 smithi123 ceph-mon[44966]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:22:10.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:10 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T14:22:10.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:10 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T14:22:10.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:10 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:22:10.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:10 smithi194 ceph-mon[45450]: from='osd.3 [v2:172.21.15.19:6810/3868219159,v1:172.21.15.19:6811/3868219159]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]': finished 2024-01-24T14:22:10.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:10 smithi194 ceph-mon[45450]: osdmap e22: 4 total, 3 up, 4 in 2024-01-24T14:22:10.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:10 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-24T14:22:10.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:10 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-24T14:22:10.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:10 smithi194 ceph-mon[45450]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:22:10.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:10 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T14:22:10.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:10 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T14:22:10.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:10 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:22:11.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:11 smithi123 ceph-mon[44966]: purged_snaps scrub starts 2024-01-24T14:22:11.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:11 smithi123 ceph-mon[44966]: purged_snaps scrub ok 2024-01-24T14:22:11.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:11 smithi123 ceph-mon[44966]: osd.3 [v2:172.21.15.19:6810/3868219159,v1:172.21.15.19:6811/3868219159] boot 2024-01-24T14:22:11.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:11 smithi123 ceph-mon[44966]: osdmap e23: 4 total, 4 up, 4 in 2024-01-24T14:22:11.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:11 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-24T14:22:11.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:11 smithi123 ceph-mon[44966]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 106 MiB used, 358 GiB / 358 GiB avail 2024-01-24T14:22:11.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:11 smithi194 ceph-mon[45450]: purged_snaps scrub starts 2024-01-24T14:22:11.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:11 smithi194 ceph-mon[45450]: purged_snaps scrub ok 2024-01-24T14:22:11.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:11 smithi194 ceph-mon[45450]: osd.3 [v2:172.21.15.19:6810/3868219159,v1:172.21.15.19:6811/3868219159] boot 2024-01-24T14:22:11.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:11 smithi194 ceph-mon[45450]: osdmap e23: 4 total, 4 up, 4 in 2024-01-24T14:22:11.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:11 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-24T14:22:11.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:11 smithi194 ceph-mon[45450]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 106 MiB used, 358 GiB / 358 GiB avail 2024-01-24T14:22:11.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:11 smithi019 ceph-mon[40663]: purged_snaps scrub starts 2024-01-24T14:22:11.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:11 smithi019 ceph-mon[40663]: purged_snaps scrub ok 2024-01-24T14:22:11.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:11 smithi019 ceph-mon[40663]: osd.3 [v2:172.21.15.19:6810/3868219159,v1:172.21.15.19:6811/3868219159] boot 2024-01-24T14:22:11.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:11 smithi019 ceph-mon[40663]: osdmap e23: 4 total, 4 up, 4 in 2024-01-24T14:22:11.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:11 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-01-24T14:22:11.987 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:11 smithi019 ceph-mon[40663]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 106 MiB used, 358 GiB / 358 GiB avail 2024-01-24T14:22:12.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:12 smithi123 ceph-mon[44966]: osdmap e24: 4 total, 4 up, 4 in 2024-01-24T14:22:12.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:12 smithi123 ceph-mon[44966]: from='client.? 172.21.15.123:0/659836739' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "029fb28f-9af5-4239-8ee5-5ac5d04cd34b"}]: dispatch 2024-01-24T14:22:12.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:12 smithi123 ceph-mon[44966]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "029fb28f-9af5-4239-8ee5-5ac5d04cd34b"}]: dispatch 2024-01-24T14:22:12.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:12 smithi123 ceph-mon[44966]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "029fb28f-9af5-4239-8ee5-5ac5d04cd34b"}]': finished 2024-01-24T14:22:12.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:12 smithi123 ceph-mon[44966]: osdmap e25: 5 total, 4 up, 5 in 2024-01-24T14:22:12.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:12 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-24T14:22:12.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:12 smithi194 ceph-mon[45450]: osdmap e24: 4 total, 4 up, 4 in 2024-01-24T14:22:12.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:12 smithi194 ceph-mon[45450]: from='client.? 172.21.15.123:0/659836739' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "029fb28f-9af5-4239-8ee5-5ac5d04cd34b"}]: dispatch 2024-01-24T14:22:12.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:12 smithi194 ceph-mon[45450]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "029fb28f-9af5-4239-8ee5-5ac5d04cd34b"}]: dispatch 2024-01-24T14:22:12.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:12 smithi194 ceph-mon[45450]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "029fb28f-9af5-4239-8ee5-5ac5d04cd34b"}]': finished 2024-01-24T14:22:12.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:12 smithi194 ceph-mon[45450]: osdmap e25: 5 total, 4 up, 5 in 2024-01-24T14:22:12.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:12 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-24T14:22:12.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:12 smithi019 ceph-mon[40663]: osdmap e24: 4 total, 4 up, 4 in 2024-01-24T14:22:12.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:12 smithi019 ceph-mon[40663]: from='client.? 172.21.15.123:0/659836739' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "029fb28f-9af5-4239-8ee5-5ac5d04cd34b"}]: dispatch 2024-01-24T14:22:12.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:12 smithi019 ceph-mon[40663]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "029fb28f-9af5-4239-8ee5-5ac5d04cd34b"}]: dispatch 2024-01-24T14:22:12.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:12 smithi019 ceph-mon[40663]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "029fb28f-9af5-4239-8ee5-5ac5d04cd34b"}]': finished 2024-01-24T14:22:12.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:12 smithi019 ceph-mon[40663]: osdmap e25: 5 total, 4 up, 5 in 2024-01-24T14:22:12.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:12 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-24T14:22:13.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:13 smithi123 ceph-mon[44966]: from='client.? 172.21.15.123:0/2117139595' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T14:22:13.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:13 smithi123 ceph-mon[44966]: osdmap e26: 5 total, 4 up, 5 in 2024-01-24T14:22:13.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:13 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-24T14:22:13.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:13 smithi123 ceph-mon[44966]: pgmap v81: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 106 MiB used, 358 GiB / 358 GiB avail; 166 KiB/s, 0 objects/s recovering 2024-01-24T14:22:13.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:13 smithi194 ceph-mon[45450]: from='client.? 172.21.15.123:0/2117139595' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T14:22:13.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:13 smithi194 ceph-mon[45450]: osdmap e26: 5 total, 4 up, 5 in 2024-01-24T14:22:13.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:13 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-24T14:22:13.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:13 smithi194 ceph-mon[45450]: pgmap v81: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 106 MiB used, 358 GiB / 358 GiB avail; 166 KiB/s, 0 objects/s recovering 2024-01-24T14:22:13.985 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:13 smithi019 ceph-mon[40663]: from='client.? 172.21.15.123:0/2117139595' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T14:22:13.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:13 smithi019 ceph-mon[40663]: osdmap e26: 5 total, 4 up, 5 in 2024-01-24T14:22:13.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:13 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-24T14:22:13.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:13 smithi019 ceph-mon[40663]: pgmap v81: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 106 MiB used, 358 GiB / 358 GiB avail; 166 KiB/s, 0 objects/s recovering 2024-01-24T14:22:15.235 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:14 smithi019 ceph-mon[40663]: osdmap e27: 5 total, 4 up, 5 in 2024-01-24T14:22:15.235 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:14 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-24T14:22:15.301 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:14 smithi123 ceph-mon[44966]: osdmap e27: 5 total, 4 up, 5 in 2024-01-24T14:22:15.302 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:14 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-24T14:22:15.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:14 smithi194 ceph-mon[45450]: osdmap e27: 5 total, 4 up, 5 in 2024-01-24T14:22:15.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:14 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-24T14:22:16.235 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:15 smithi019 ceph-mon[40663]: pgmap v83: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 106 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-01-24T14:22:16.276 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:15 smithi123 ceph-mon[44966]: pgmap v83: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 106 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-01-24T14:22:16.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:15 smithi194 ceph-mon[45450]: pgmap v83: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 106 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-01-24T14:22:18.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:17 smithi123 ceph-mon[44966]: pgmap v84: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-01-24T14:22:18.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:17 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-01-24T14:22:18.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:17 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:22:18.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:17 smithi194 ceph-mon[45450]: pgmap v84: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-01-24T14:22:18.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:17 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-01-24T14:22:18.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:17 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:22:18.485 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:17 smithi019 ceph-mon[40663]: pgmap v84: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-01-24T14:22:18.485 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:17 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-01-24T14:22:18.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:17 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:22:19.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:18 smithi123 ceph-mon[44966]: Deploying daemon osd.4 on smithi123 2024-01-24T14:22:19.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:18 smithi194 ceph-mon[45450]: Deploying daemon osd.4 on smithi123 2024-01-24T14:22:19.485 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:18 smithi019 ceph-mon[40663]: Deploying daemon osd.4 on smithi123 2024-01-24T14:22:20.727 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:20 smithi123 ceph-mon[44966]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-01-24T14:22:20.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:20 smithi019 ceph-mon[40663]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-01-24T14:22:20.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:20 smithi194 ceph-mon[45450]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-01-24T14:22:21.713 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:21 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:22:21.713 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:21 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:21.713 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:21 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:21.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:21 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:22:21.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:21 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:21.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:21 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:21.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:21 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:22:21.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:21 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:21.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:21 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:22.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:22 smithi123 ceph-mon[44966]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-01-24T14:22:22.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:22 smithi019 ceph-mon[40663]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-01-24T14:22:22.843 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:22 smithi194 ceph-mon[45450]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-01-24T14:22:23.533 INFO:teuthology.orchestra.run.smithi123.stdout:Created osd(s) 4 on host 'smithi123' 2024-01-24T14:22:23.534 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:23.532+0000 7f1ba9ffb700 1 -- 172.21.15.123:0/4226335993 <== mgr.14152 v2:172.21.15.19:6800/1634350318 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f1bac061180 con 0x7f1b9c0a05a0 2024-01-24T14:22:23.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:23.534+0000 7f1bb3207700 1 -- 172.21.15.123:0/4226335993 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f1b9c0a05a0 msgr2=0x7f1b9c0a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:22:23.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:23.534+0000 7f1bb3207700 1 --2- 172.21.15.123:0/4226335993 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f1b9c0a05a0 0x7f1b9c0a2a60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1ba0005d90 tx=0x7f1ba0005d20 comp rx=0 tx=0).stop 2024-01-24T14:22:23.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:23.534+0000 7f1bb3207700 1 -- 172.21.15.123:0/4226335993 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1bac104230 msgr2=0x7f1bac197120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:22:23.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:23.534+0000 7f1bb3207700 1 --2- 172.21.15.123:0/4226335993 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1bac104230 0x7f1bac197120 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1b9800b530 tx=0x7f1b9800b840 comp rx=0 tx=0).stop 2024-01-24T14:22:23.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:23.534+0000 7f1bb3207700 1 -- 172.21.15.123:0/4226335993 shutdown_connections 2024-01-24T14:22:23.535 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:23.534+0000 7f1bb3207700 1 --2- 172.21.15.123:0/4226335993 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f1b9c0a05a0 0x7f1b9c0a2a60 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:22:23.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:23.534+0000 7f1bb3207700 1 --2- 172.21.15.123:0/4226335993 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1bac106b60 0x7f1bac19e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:22:23.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:23.534+0000 7f1bb3207700 1 --2- 172.21.15.123:0/4226335993 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1bac101900 0x7f1bac196be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:22:23.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:23.534+0000 7f1bb3207700 1 --2- 172.21.15.123:0/4226335993 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1bac104230 0x7f1bac197120 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:22:23.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:23.534+0000 7f1bb3207700 1 -- 172.21.15.123:0/4226335993 >> 172.21.15.123:0/4226335993 conn(0x7f1bac0fb250 msgr2=0x7f1bac102570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:22:23.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:23.534+0000 7f1bb3207700 1 -- 172.21.15.123:0/4226335993 shutdown_connections 2024-01-24T14:22:23.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:22:23.534+0000 7f1bb3207700 1 -- 172.21.15.123:0/4226335993 wait complete. 2024-01-24T14:22:23.951 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:23 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:23.951 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:23 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:23.951 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:23 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:22:23.951 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:23 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:22:23.951 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:23 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:23.951 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:23 smithi123 ceph-mon[44966]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-01-24T14:22:23.951 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:23 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:22:23.951 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:23 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:23.951 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:23 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:23.952 DEBUG:teuthology.orchestra.run.smithi123:osd.4> sudo journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@osd.4.service 2024-01-24T14:22:23.954 INFO:tasks.cephadm:Deploying osd.5 on smithi194 with /dev/vg_nvme/lv_3... 2024-01-24T14:22:23.954 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-01-24T14:22:23.985 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:23 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:23.985 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:23 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:23.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:23 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:22:23.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:23 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:22:23.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:23 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:23.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:23 smithi019 ceph-mon[40663]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-01-24T14:22:23.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:23 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:22:23.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:23 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:23.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:23 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:23.986 INFO:journalctl@ceph.osd.4.smithi123.stdout:-- Logs begin at Wed 2024-01-24 14:03:45 UTC. -- 2024-01-24T14:22:23.988 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:23 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:23.988 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:23 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:23.988 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:23 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:22:23.988 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:23 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:22:23.989 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:23 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:23.989 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:23 smithi194 ceph-mon[45450]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-01-24T14:22:23.989 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:23 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:22:23.989 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:23 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:23.989 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:23 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:24.186 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/mon.c/config 2024-01-24T14:22:25.381 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-01-24T14:22:25.381 INFO:teuthology.orchestra.run.smithi194.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-01-24T14:22:25.381 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10+0 records in 2024-01-24T14:22:25.381 INFO:teuthology.orchestra.run.smithi194.stderr:10+0 records out 2024-01-24T14:22:25.381 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0157474 s, 666 MB/s 2024-01-24T14:22:25.381 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping successful for: 2024-01-24T14:22:25.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:25 smithi123 ceph-mon[44966]: from='osd.4 [v2:172.21.15.123:6808/3611579809,v1:172.21.15.123:6809/3611579809]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-01-24T14:22:25.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:25 smithi123 ceph-mon[44966]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-01-24T14:22:25.550 INFO:journalctl@ceph.osd.4.smithi123.stdout:Jan 24 14:22:25 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-4[58441]: 2024-01-24T14:22:25.221+0000 7f0152ca1640 -1 osd.4 0 log_to_monitors true 2024-01-24T14:22:25.648 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:25 smithi194 ceph-mon[45450]: from='osd.4 [v2:172.21.15.123:6808/3611579809,v1:172.21.15.123:6809/3611579809]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-01-24T14:22:25.648 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:25 smithi194 ceph-mon[45450]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-01-24T14:22:25.712 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 -- ceph orch daemon add osd smithi194:vg_nvme/lv_3 2024-01-24T14:22:25.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:25 smithi019 ceph-mon[40663]: from='osd.4 [v2:172.21.15.123:6808/3611579809,v1:172.21.15.123:6809/3611579809]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-01-24T14:22:25.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:25 smithi019 ceph-mon[40663]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-01-24T14:22:25.972 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/mon.c/config 2024-01-24T14:22:26.695 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:26 smithi194 ceph-mon[45450]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-01-24T14:22:26.695 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:26 smithi194 ceph-mon[45450]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-01-24T14:22:26.695 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:26 smithi194 ceph-mon[45450]: osdmap e28: 5 total, 4 up, 5 in 2024-01-24T14:22:26.695 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:26 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-24T14:22:26.695 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:26 smithi194 ceph-mon[45450]: from='osd.4 [v2:172.21.15.123:6808/3611579809,v1:172.21.15.123:6809/3611579809]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-01-24T14:22:26.695 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:26 smithi194 ceph-mon[45450]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-01-24T14:22:26.695 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:26 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:26.695 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:26 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:26.695 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:26 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:22:26.695 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:26 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:22:26.695 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:26 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:26.695 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:26 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:22:26.696 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:26 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:22:26.696 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:26 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:26.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:26 smithi019 ceph-mon[40663]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-01-24T14:22:26.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:26 smithi019 ceph-mon[40663]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-01-24T14:22:26.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:26 smithi019 ceph-mon[40663]: osdmap e28: 5 total, 4 up, 5 in 2024-01-24T14:22:26.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:26 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-24T14:22:26.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:26 smithi019 ceph-mon[40663]: from='osd.4 [v2:172.21.15.123:6808/3611579809,v1:172.21.15.123:6809/3611579809]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-01-24T14:22:26.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:26 smithi019 ceph-mon[40663]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-01-24T14:22:26.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:26 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:26.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:26 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:26.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:26 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:22:26.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:26 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:22:26.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:26 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:26.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:26 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:22:26.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:26 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:22:26.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:26 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:26.799 INFO:journalctl@ceph.osd.4.smithi123.stdout:Jan 24 14:22:26 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-4[58441]: 2024-01-24T14:22:26.413+0000 7f0148306700 -1 osd.4 0 waiting for initial osdmap 2024-01-24T14:22:26.799 INFO:journalctl@ceph.osd.4.smithi123.stdout:Jan 24 14:22:26 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-4[58441]: 2024-01-24T14:22:26.418+0000 7f01420f7700 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-24T14:22:26.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:26 smithi123 ceph-mon[44966]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-01-24T14:22:26.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:26 smithi123 ceph-mon[44966]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-01-24T14:22:26.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:26 smithi123 ceph-mon[44966]: osdmap e28: 5 total, 4 up, 5 in 2024-01-24T14:22:26.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:26 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-24T14:22:26.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:26 smithi123 ceph-mon[44966]: from='osd.4 [v2:172.21.15.123:6808/3611579809,v1:172.21.15.123:6809/3611579809]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-01-24T14:22:26.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:26 smithi123 ceph-mon[44966]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-01-24T14:22:26.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:26 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:26.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:26 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:26.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:26 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:22:26.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:26 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:22:26.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:26 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:26.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:26 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:22:26.801 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:26 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:22:26.801 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:26 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:26.856 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:26.855+0000 7fe96c4ba700 1 -- 172.21.15.194:0/536719428 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe964101900 msgr2=0x7fe964103cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:22:26.856 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:26.855+0000 7fe96c4ba700 1 --2- 172.21.15.194:0/536719428 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe964101900 0x7fe964103cf0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe954009990 tx=0x7fe954009ca0 comp rx=0 tx=0).stop 2024-01-24T14:22:26.856 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:26.855+0000 7fe96c4ba700 1 -- 172.21.15.194:0/536719428 shutdown_connections 2024-01-24T14:22:26.856 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:26.855+0000 7fe96c4ba700 1 --2- 172.21.15.194:0/536719428 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe964106b60 0x7fe964108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:22:26.856 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:26.855+0000 7fe96c4ba700 1 --2- 172.21.15.194:0/536719428 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe964101900 0x7fe964103cf0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:22:26.856 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:26.855+0000 7fe96c4ba700 1 --2- 172.21.15.194:0/536719428 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe964104230 0x7fe964106620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:22:26.856 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:26.855+0000 7fe96c4ba700 1 -- 172.21.15.194:0/536719428 >> 172.21.15.194:0/536719428 conn(0x7fe9640fb250 msgr2=0x7fe9640fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:22:26.856 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:26.855+0000 7fe96c4ba700 1 -- 172.21.15.194:0/536719428 shutdown_connections 2024-01-24T14:22:26.856 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:26.856+0000 7fe96c4ba700 1 -- 172.21.15.194:0/536719428 wait complete. 2024-01-24T14:22:26.858 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:26.856+0000 7fe96c4ba700 1 Processor -- start 2024-01-24T14:22:26.858 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:26.856+0000 7fe96c4ba700 1 -- start start 2024-01-24T14:22:26.858 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:26.856+0000 7fe96c4ba700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe964101900 0x7fe964196aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:22:26.858 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:26.856+0000 7fe96c4ba700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe964104230 0x7fe964196fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:22:26.859 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:26.856+0000 7fe96c4ba700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe964106b60 0x7fe96419e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:22:26.859 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:26.856+0000 7fe96c4ba700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe96419e630 con 0x7fe964104230 2024-01-24T14:22:26.859 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:26.856+0000 7fe96c4ba700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe96419e770 con 0x7fe964101900 2024-01-24T14:22:26.859 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:26.856+0000 7fe96c4ba700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe96419e900 con 0x7fe964106b60 2024-01-24T14:22:26.859 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:26.856+0000 7fe96aa57700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe964106b60 0x7fe96419e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:22:26.859 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:26.856+0000 7fe96aa57700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe964106b60 0x7fe96419e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:51018/0 (socket says 172.21.15.194:51018) 2024-01-24T14:22:26.859 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:26.856+0000 7fe96aa57700 1 -- 172.21.15.194:0/3219947182 learned_addr learned my addr 172.21.15.194:0/3219947182 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-24T14:22:26.859 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:26.856+0000 7fe969a55700 1 --2- 172.21.15.194:0/3219947182 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe964104230 0x7fe964196fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:22:26.859 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:26.856+0000 7fe96a256700 1 --2- 172.21.15.194:0/3219947182 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe964101900 0x7fe964196aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:22:26.859 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:26.857+0000 7fe96aa57700 1 -- 172.21.15.194:0/3219947182 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe964101900 msgr2=0x7fe964196aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:22:26.859 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:26.857+0000 7fe96aa57700 1 --2- 172.21.15.194:0/3219947182 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe964101900 0x7fe964196aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:22:26.859 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:26.857+0000 7fe96aa57700 1 -- 172.21.15.194:0/3219947182 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe964104230 msgr2=0x7fe964196fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:22:26.859 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:26.857+0000 7fe96aa57700 1 --2- 172.21.15.194:0/3219947182 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe964104230 0x7fe964196fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:22:26.860 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:26.857+0000 7fe96aa57700 1 -- 172.21.15.194:0/3219947182 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9540095f0 con 0x7fe964106b60 2024-01-24T14:22:26.860 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:26.857+0000 7fe96aa57700 1 --2- 172.21.15.194:0/3219947182 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe964106b60 0x7fe96419e060 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe95c007ee0 tx=0x7fe95c00d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:22:26.860 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:26.857+0000 7fe95b7fe700 1 -- 172.21.15.194:0/3219947182 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe95c004db0 con 0x7fe964106b60 2024-01-24T14:22:26.860 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:26.857+0000 7fe95b7fe700 1 -- 172.21.15.194:0/3219947182 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe95c00f040 con 0x7fe964106b60 2024-01-24T14:22:26.860 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:26.857+0000 7fe96c4ba700 1 -- 172.21.15.194:0/3219947182 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe96419ebe0 con 0x7fe964106b60 2024-01-24T14:22:26.860 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:26.857+0000 7fe95b7fe700 1 -- 172.21.15.194:0/3219947182 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe95c005920 con 0x7fe964106b60 2024-01-24T14:22:26.860 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:26.857+0000 7fe96c4ba700 1 -- 172.21.15.194:0/3219947182 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe96419f0e0 con 0x7fe964106b60 2024-01-24T14:22:26.860 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:26.858+0000 7fe96c4ba700 1 -- 172.21.15.194:0/3219947182 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9640fcdd0 con 0x7fe964106b60 2024-01-24T14:22:26.861 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:26.859+0000 7fe95b7fe700 1 -- 172.21.15.194:0/3219947182 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7fe95c00db20 con 0x7fe964106b60 2024-01-24T14:22:26.861 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:26.860+0000 7fe95b7fe700 1 --2- 172.21.15.194:0/3219947182 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fe9500a05a0 0x7fe9500a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:22:26.861 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:26.860+0000 7fe95b7fe700 1 -- 172.21.15.194:0/3219947182 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 3821+0+0 (secure 0 0 0) 0x7fe95c0ca080 con 0x7fe964106b60 2024-01-24T14:22:26.861 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:26.860+0000 7fe96a256700 1 --2- 172.21.15.194:0/3219947182 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fe9500a05a0 0x7fe9500a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:22:26.861 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:26.860+0000 7fe96a256700 1 --2- 172.21.15.194:0/3219947182 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fe9500a05a0 0x7fe9500a2a60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe95400b5c0 tx=0x7fe954005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:22:26.864 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:26.864+0000 7fe95b7fe700 1 -- 172.21.15.194:0/3219947182 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fe95c094db0 con 0x7fe964106b60 2024-01-24T14:22:27.057 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:27.057+0000 7fe96c4ba700 1 -- 172.21.15.194:0/3219947182 --> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fe964061180 con 0x7fe9500a05a0 2024-01-24T14:22:27.703 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:27 smithi194 ceph-mon[45450]: Detected new or changed devices on smithi123 2024-01-24T14:22:27.703 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:27 smithi194 ceph-mon[45450]: Adjusting osd_memory_target on smithi123 to 8557M 2024-01-24T14:22:27.703 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:27 smithi194 ceph-mon[45450]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-01-24T14:22:27.703 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:27 smithi194 ceph-mon[45450]: osdmap e29: 5 total, 4 up, 5 in 2024-01-24T14:22:27.703 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:27 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-24T14:22:27.703 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:27 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-24T14:22:27.703 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:27 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T14:22:27.703 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:27 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T14:22:27.703 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:27 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:22:27.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:27 smithi019 ceph-mon[40663]: Detected new or changed devices on smithi123 2024-01-24T14:22:27.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:27 smithi019 ceph-mon[40663]: Adjusting osd_memory_target on smithi123 to 8557M 2024-01-24T14:22:27.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:27 smithi019 ceph-mon[40663]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-01-24T14:22:27.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:27 smithi019 ceph-mon[40663]: osdmap e29: 5 total, 4 up, 5 in 2024-01-24T14:22:27.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:27 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-24T14:22:27.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:27 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-24T14:22:27.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:27 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T14:22:27.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:27 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T14:22:27.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:27 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:22:27.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:27 smithi123 ceph-mon[44966]: Detected new or changed devices on smithi123 2024-01-24T14:22:27.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:27 smithi123 ceph-mon[44966]: Adjusting osd_memory_target on smithi123 to 8557M 2024-01-24T14:22:27.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:27 smithi123 ceph-mon[44966]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-01-24T14:22:27.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:27 smithi123 ceph-mon[44966]: osdmap e29: 5 total, 4 up, 5 in 2024-01-24T14:22:27.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:27 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-24T14:22:27.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:27 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-24T14:22:27.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:27 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T14:22:27.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:27 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T14:22:27.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:27 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:22:28.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:28 smithi019 ceph-mon[40663]: purged_snaps scrub starts 2024-01-24T14:22:28.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:28 smithi019 ceph-mon[40663]: purged_snaps scrub ok 2024-01-24T14:22:28.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:28 smithi019 ceph-mon[40663]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-01-24T14:22:28.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:28 smithi019 ceph-mon[40663]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:22:28.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:28 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-24T14:22:28.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:28 smithi019 ceph-mon[40663]: osd.4 [v2:172.21.15.123:6808/3611579809,v1:172.21.15.123:6809/3611579809] boot 2024-01-24T14:22:28.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:28 smithi019 ceph-mon[40663]: osdmap e30: 5 total, 5 up, 5 in 2024-01-24T14:22:28.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:28 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-24T14:22:28.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:28 smithi123 ceph-mon[44966]: purged_snaps scrub starts 2024-01-24T14:22:28.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:28 smithi123 ceph-mon[44966]: purged_snaps scrub ok 2024-01-24T14:22:28.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:28 smithi123 ceph-mon[44966]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-01-24T14:22:28.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:28 smithi123 ceph-mon[44966]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:22:28.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:28 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-24T14:22:28.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:28 smithi123 ceph-mon[44966]: osd.4 [v2:172.21.15.123:6808/3611579809,v1:172.21.15.123:6809/3611579809] boot 2024-01-24T14:22:28.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:28 smithi123 ceph-mon[44966]: osdmap e30: 5 total, 5 up, 5 in 2024-01-24T14:22:28.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:28 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-24T14:22:28.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:28 smithi194 ceph-mon[45450]: purged_snaps scrub starts 2024-01-24T14:22:28.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:28 smithi194 ceph-mon[45450]: purged_snaps scrub ok 2024-01-24T14:22:28.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:28 smithi194 ceph-mon[45450]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-01-24T14:22:28.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:28 smithi194 ceph-mon[45450]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:22:28.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:28 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-24T14:22:28.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:28 smithi194 ceph-mon[45450]: osd.4 [v2:172.21.15.123:6808/3611579809,v1:172.21.15.123:6809/3611579809] boot 2024-01-24T14:22:28.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:28 smithi194 ceph-mon[45450]: osdmap e30: 5 total, 5 up, 5 in 2024-01-24T14:22:28.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:28 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-01-24T14:22:29.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:29 smithi019 ceph-mon[40663]: osdmap e31: 5 total, 5 up, 5 in 2024-01-24T14:22:29.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:29 smithi019 ceph-mon[40663]: from='client.? 172.21.15.194:0/1451561447' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e97f23b-f803-4fed-a0a2-7aee7de4970f"}]: dispatch 2024-01-24T14:22:29.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:29 smithi019 ceph-mon[40663]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e97f23b-f803-4fed-a0a2-7aee7de4970f"}]: dispatch 2024-01-24T14:22:29.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:29 smithi019 ceph-mon[40663]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6e97f23b-f803-4fed-a0a2-7aee7de4970f"}]': finished 2024-01-24T14:22:29.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:29 smithi019 ceph-mon[40663]: osdmap e32: 6 total, 5 up, 6 in 2024-01-24T14:22:29.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:29 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-24T14:22:29.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:29 smithi123 ceph-mon[44966]: osdmap e31: 5 total, 5 up, 5 in 2024-01-24T14:22:29.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:29 smithi123 ceph-mon[44966]: from='client.? 172.21.15.194:0/1451561447' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e97f23b-f803-4fed-a0a2-7aee7de4970f"}]: dispatch 2024-01-24T14:22:29.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:29 smithi123 ceph-mon[44966]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e97f23b-f803-4fed-a0a2-7aee7de4970f"}]: dispatch 2024-01-24T14:22:29.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:29 smithi123 ceph-mon[44966]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6e97f23b-f803-4fed-a0a2-7aee7de4970f"}]': finished 2024-01-24T14:22:29.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:29 smithi123 ceph-mon[44966]: osdmap e32: 6 total, 5 up, 6 in 2024-01-24T14:22:29.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:29 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-24T14:22:29.843 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:29 smithi194 ceph-mon[45450]: osdmap e31: 5 total, 5 up, 5 in 2024-01-24T14:22:29.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:29 smithi194 ceph-mon[45450]: from='client.? 172.21.15.194:0/1451561447' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e97f23b-f803-4fed-a0a2-7aee7de4970f"}]: dispatch 2024-01-24T14:22:29.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:29 smithi194 ceph-mon[45450]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e97f23b-f803-4fed-a0a2-7aee7de4970f"}]: dispatch 2024-01-24T14:22:29.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:29 smithi194 ceph-mon[45450]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6e97f23b-f803-4fed-a0a2-7aee7de4970f"}]': finished 2024-01-24T14:22:29.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:29 smithi194 ceph-mon[45450]: osdmap e32: 6 total, 5 up, 6 in 2024-01-24T14:22:29.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:29 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-24T14:22:30.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:30 smithi019 ceph-mon[40663]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-01-24T14:22:30.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:30 smithi019 ceph-mon[40663]: from='client.? 172.21.15.194:0/389986168' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T14:22:30.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:30 smithi123 ceph-mon[44966]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-01-24T14:22:30.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:30 smithi123 ceph-mon[44966]: from='client.? 172.21.15.194:0/389986168' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T14:22:30.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:30 smithi194 ceph-mon[45450]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-01-24T14:22:30.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:30 smithi194 ceph-mon[45450]: from='client.? 172.21.15.194:0/389986168' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T14:22:32.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:32 smithi019 ceph-mon[40663]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-01-24T14:22:32.781 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:32 smithi194 ceph-mon[45450]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-01-24T14:22:32.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:32 smithi123 ceph-mon[44966]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-01-24T14:22:34.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:34 smithi194 ceph-mon[45450]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-01-24T14:22:34.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:34 smithi019 ceph-mon[40663]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-01-24T14:22:34.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:34 smithi123 ceph-mon[44966]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-01-24T14:22:35.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:35 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-01-24T14:22:35.722 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:35 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:22:35.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:35 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-01-24T14:22:35.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:35 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:22:35.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:35 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-01-24T14:22:35.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:35 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:22:36.718 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:36 smithi194 ceph-mon[45450]: Deploying daemon osd.5 on smithi194 2024-01-24T14:22:36.719 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:36 smithi194 ceph-mon[45450]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-01-24T14:22:36.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:36 smithi019 ceph-mon[40663]: Deploying daemon osd.5 on smithi194 2024-01-24T14:22:36.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:36 smithi019 ceph-mon[40663]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-01-24T14:22:36.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:36 smithi123 ceph-mon[44966]: Deploying daemon osd.5 on smithi194 2024-01-24T14:22:36.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:36 smithi123 ceph-mon[44966]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-01-24T14:22:38.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:38 smithi019 ceph-mon[40663]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-01-24T14:22:38.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:38 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:38.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:38 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:38.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:38 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:22:38.758 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:38 smithi194 ceph-mon[45450]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-01-24T14:22:38.758 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:38 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:38.758 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:38 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:38.758 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:38 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:22:38.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:38 smithi123 ceph-mon[44966]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-01-24T14:22:38.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:38 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:38.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:38 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:38.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:38 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:22:40.365 INFO:teuthology.orchestra.run.smithi194.stdout:Created osd(s) 5 on host 'smithi194' 2024-01-24T14:22:40.365 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:40.364+0000 7fe95b7fe700 1 -- 172.21.15.194:0/3219947182 <== mgr.14152 v2:172.21.15.19:6800/1634350318 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe964061180 con 0x7fe9500a05a0 2024-01-24T14:22:40.367 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:40.367+0000 7fe96c4ba700 1 -- 172.21.15.194:0/3219947182 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fe9500a05a0 msgr2=0x7fe9500a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:22:40.367 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:40.367+0000 7fe96c4ba700 1 --2- 172.21.15.194:0/3219947182 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fe9500a05a0 0x7fe9500a2a60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe95400b5c0 tx=0x7fe954005fd0 comp rx=0 tx=0).stop 2024-01-24T14:22:40.367 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:40.367+0000 7fe96c4ba700 1 -- 172.21.15.194:0/3219947182 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe964106b60 msgr2=0x7fe96419e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:22:40.367 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:40.367+0000 7fe96c4ba700 1 --2- 172.21.15.194:0/3219947182 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe964106b60 0x7fe96419e060 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe95c007ee0 tx=0x7fe95c00d3b0 comp rx=0 tx=0).stop 2024-01-24T14:22:40.367 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:40.367+0000 7fe96c4ba700 1 -- 172.21.15.194:0/3219947182 shutdown_connections 2024-01-24T14:22:40.367 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:40.367+0000 7fe96c4ba700 1 --2- 172.21.15.194:0/3219947182 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe964106b60 0x7fe96419e060 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:22:40.367 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:40.367+0000 7fe96c4ba700 1 --2- 172.21.15.194:0/3219947182 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe964101900 0x7fe964196aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:22:40.367 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:40.367+0000 7fe96c4ba700 1 --2- 172.21.15.194:0/3219947182 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fe9500a05a0 0x7fe9500a2a60 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:22:40.367 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:40.367+0000 7fe96c4ba700 1 --2- 172.21.15.194:0/3219947182 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe964104230 0x7fe964196fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:22:40.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:40.367+0000 7fe96c4ba700 1 -- 172.21.15.194:0/3219947182 >> 172.21.15.194:0/3219947182 conn(0x7fe9640fb250 msgr2=0x7fe964102570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:22:40.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:40.367+0000 7fe96c4ba700 1 -- 172.21.15.194:0/3219947182 shutdown_connections 2024-01-24T14:22:40.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:22:40.367+0000 7fe96c4ba700 1 -- 172.21.15.194:0/3219947182 wait complete. 2024-01-24T14:22:40.534 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:40 smithi194 ceph-mon[45450]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-01-24T14:22:40.535 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:40 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:40.535 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:40 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:40.535 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:40 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:22:40.535 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:40 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:22:40.535 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:40 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:40.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:40 smithi123 ceph-mon[44966]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-01-24T14:22:40.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:40 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:40.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:40 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:40.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:40 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:22:40.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:40 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:22:40.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:40 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:40.713 DEBUG:teuthology.orchestra.run.smithi194:osd.5> sudo journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@osd.5.service 2024-01-24T14:22:40.715 INFO:tasks.cephadm:Deploying osd.6 on smithi019 with /dev/vg_nvme/lv_2... 2024-01-24T14:22:40.716 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-01-24T14:22:40.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:40 smithi019 ceph-mon[40663]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-01-24T14:22:40.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:40 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:40.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:40 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:40.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:40 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:22:40.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:40 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:22:40.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:40 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:40.746 INFO:journalctl@ceph.osd.5.smithi194.stdout:-- Logs begin at Wed 2024-01-24 14:04:01 UTC. -- 2024-01-24T14:22:40.969 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/mon.a/config 2024-01-24T14:22:41.354 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:41 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:41.354 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:41 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:41.354 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:41 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:22:41.558 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:41 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:41.558 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:41 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:41.559 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:41 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:22:41.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:41 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:41.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:41 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:41.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:41 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:22:42.408 INFO:teuthology.orchestra.run.smithi019.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-01-24T14:22:42.408 INFO:teuthology.orchestra.run.smithi019.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-01-24T14:22:42.408 INFO:teuthology.orchestra.run.smithi019.stderr: stderr: 10+0 records in 2024-01-24T14:22:42.408 INFO:teuthology.orchestra.run.smithi019.stderr:10+0 records out 2024-01-24T14:22:42.409 INFO:teuthology.orchestra.run.smithi019.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0140318 s, 747 MB/s 2024-01-24T14:22:42.409 INFO:teuthology.orchestra.run.smithi019.stderr:--> Zapping successful for: 2024-01-24T14:22:42.525 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jan 24 14:22:42 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-5[59863]: 2024-01-24T14:22:42.156+0000 7f9982b71640 -1 osd.5 0 log_to_monitors true 2024-01-24T14:22:42.526 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:42 smithi194 ceph-mon[45450]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-01-24T14:22:42.526 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:42 smithi194 ceph-mon[45450]: from='osd.5 [v2:172.21.15.194:6808/2483361871,v1:172.21.15.194:6809/2483361871]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-01-24T14:22:42.526 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:42 smithi194 ceph-mon[45450]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-01-24T14:22:42.611 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:42 smithi019 ceph-mon[40663]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-01-24T14:22:42.611 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:42 smithi019 ceph-mon[40663]: from='osd.5 [v2:172.21.15.194:6808/2483361871,v1:172.21.15.194:6809/2483361871]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-01-24T14:22:42.612 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:42 smithi019 ceph-mon[40663]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-01-24T14:22:42.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:42 smithi123 ceph-mon[44966]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-01-24T14:22:42.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:42 smithi123 ceph-mon[44966]: from='osd.5 [v2:172.21.15.194:6808/2483361871,v1:172.21.15.194:6809/2483361871]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-01-24T14:22:42.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:42 smithi123 ceph-mon[44966]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-01-24T14:22:42.830 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 -- ceph orch daemon add osd smithi019:vg_nvme/lv_2 2024-01-24T14:22:43.121 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/mon.a/config 2024-01-24T14:22:43.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:43 smithi019 ceph-mon[40663]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-01-24T14:22:43.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:43 smithi019 ceph-mon[40663]: osdmap e33: 6 total, 5 up, 6 in 2024-01-24T14:22:43.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:43 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-24T14:22:43.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:43 smithi019 ceph-mon[40663]: from='osd.5 [v2:172.21.15.194:6808/2483361871,v1:172.21.15.194:6809/2483361871]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-01-24T14:22:43.487 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:43 smithi019 ceph-mon[40663]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-01-24T14:22:43.487 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:43 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:43.487 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:43 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:43.487 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:43 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:22:43.487 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:43 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:22:43.487 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:43 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:43.487 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:43 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:22:43.487 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:43 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:22:43.487 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:43 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:43.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:43 smithi123 ceph-mon[44966]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-01-24T14:22:43.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:43 smithi123 ceph-mon[44966]: osdmap e33: 6 total, 5 up, 6 in 2024-01-24T14:22:43.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:43 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-24T14:22:43.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:43 smithi123 ceph-mon[44966]: from='osd.5 [v2:172.21.15.194:6808/2483361871,v1:172.21.15.194:6809/2483361871]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-01-24T14:22:43.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:43 smithi123 ceph-mon[44966]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-01-24T14:22:43.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:43 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:43.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:43 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:43.801 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:43 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:22:43.801 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:43 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:22:43.801 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:43 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:43.801 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:43 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:22:43.801 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:43 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:22:43.801 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:43 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:43.844 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jan 24 14:22:43 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-5[59863]: 2024-01-24T14:22:43.349+0000 7f99781d6700 -1 osd.5 0 waiting for initial osdmap 2024-01-24T14:22:43.844 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jan 24 14:22:43 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-5[59863]: 2024-01-24T14:22:43.356+0000 7f99707c4700 -1 osd.5 34 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-24T14:22:43.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:43 smithi194 ceph-mon[45450]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-01-24T14:22:43.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:43 smithi194 ceph-mon[45450]: osdmap e33: 6 total, 5 up, 6 in 2024-01-24T14:22:43.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:43 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-24T14:22:43.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:43 smithi194 ceph-mon[45450]: from='osd.5 [v2:172.21.15.194:6808/2483361871,v1:172.21.15.194:6809/2483361871]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-01-24T14:22:43.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:43 smithi194 ceph-mon[45450]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-01-24T14:22:43.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:43 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:43.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:43 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:43.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:43 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:22:43.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:43 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:22:43.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:43 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:43.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:43 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:22:43.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:43 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:22:43.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:43 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:44.072 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:44.071+0000 7fd341766700 1 -- 172.21.15.19:0/702089815 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd33c1041c0 msgr2=0x7fd33c1065b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:22:44.072 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:44.071+0000 7fd341766700 1 --2- 172.21.15.19:0/702089815 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd33c1041c0 0x7fd33c1065b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd330009990 tx=0x7fd330009ca0 comp rx=0 tx=0).stop 2024-01-24T14:22:44.072 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:44.071+0000 7fd341766700 1 -- 172.21.15.19:0/702089815 shutdown_connections 2024-01-24T14:22:44.072 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:44.071+0000 7fd341766700 1 --2- 172.21.15.19:0/702089815 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd33c106af0 0x7fd33c108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:22:44.072 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:44.071+0000 7fd341766700 1 --2- 172.21.15.19:0/702089815 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd33c101890 0x7fd33c103c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:22:44.072 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:44.071+0000 7fd341766700 1 --2- 172.21.15.19:0/702089815 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd33c1041c0 0x7fd33c1065b0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:22:44.072 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:44.071+0000 7fd341766700 1 -- 172.21.15.19:0/702089815 >> 172.21.15.19:0/702089815 conn(0x7fd33c0fb2c0 msgr2=0x7fd33c0fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:22:44.073 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:44.071+0000 7fd341766700 1 -- 172.21.15.19:0/702089815 shutdown_connections 2024-01-24T14:22:44.073 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:44.071+0000 7fd341766700 1 -- 172.21.15.19:0/702089815 wait complete. 2024-01-24T14:22:44.074 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:44.072+0000 7fd341766700 1 Processor -- start 2024-01-24T14:22:44.074 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:44.072+0000 7fd341766700 1 -- start start 2024-01-24T14:22:44.075 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:44.072+0000 7fd341766700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd33c101890 0x7fd33c1011f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:22:44.075 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:44.072+0000 7fd341766700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd33c1041c0 0x7fd33c0ff840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:22:44.075 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:44.072+0000 7fd341766700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd33c106af0 0x7fd33c0ffd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:22:44.075 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:44.072+0000 7fd341766700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd33c100500 con 0x7fd33c1041c0 2024-01-24T14:22:44.075 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:44.072+0000 7fd341766700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd33c100670 con 0x7fd33c106af0 2024-01-24T14:22:44.075 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:44.072+0000 7fd341766700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd33c1a0b50 con 0x7fd33c101890 2024-01-24T14:22:44.075 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:44.072+0000 7fd33a7fc700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd33c1041c0 0x7fd33c0ff840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:22:44.075 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:44.072+0000 7fd33a7fc700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd33c1041c0 0x7fd33c0ff840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:33134/0 (socket says 172.21.15.19:33134) 2024-01-24T14:22:44.075 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:44.072+0000 7fd33a7fc700 1 -- 172.21.15.19:0/728629055 learned_addr learned my addr 172.21.15.19:0/728629055 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:22:44.075 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:44.072+0000 7fd33affd700 1 --2- 172.21.15.19:0/728629055 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd33c101890 0x7fd33c1011f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:22:44.075 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:44.072+0000 7fd33b7fe700 1 --2- 172.21.15.19:0/728629055 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd33c106af0 0x7fd33c0ffd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:22:44.075 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:44.073+0000 7fd33a7fc700 1 -- 172.21.15.19:0/728629055 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd33c101890 msgr2=0x7fd33c1011f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:22:44.075 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:44.073+0000 7fd33a7fc700 1 --2- 172.21.15.19:0/728629055 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd33c101890 0x7fd33c1011f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:22:44.075 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:44.073+0000 7fd33a7fc700 1 -- 172.21.15.19:0/728629055 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd33c106af0 msgr2=0x7fd33c0ffd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:22:44.075 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:44.073+0000 7fd33a7fc700 1 --2- 172.21.15.19:0/728629055 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd33c106af0 0x7fd33c0ffd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:22:44.076 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:44.073+0000 7fd33a7fc700 1 -- 172.21.15.19:0/728629055 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3300095f0 con 0x7fd33c1041c0 2024-01-24T14:22:44.076 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:44.073+0000 7fd33a7fc700 1 --2- 172.21.15.19:0/728629055 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd33c1041c0 0x7fd33c0ff840 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd330000c00 tx=0x7fd33000f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:22:44.076 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:44.073+0000 7fd323fff700 1 -- 172.21.15.19:0/728629055 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33001d070 con 0x7fd33c1041c0 2024-01-24T14:22:44.076 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:44.073+0000 7fd323fff700 1 -- 172.21.15.19:0/728629055 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd33000fb50 con 0x7fd33c1041c0 2024-01-24T14:22:44.076 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:44.073+0000 7fd341766700 1 -- 172.21.15.19:0/728629055 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd33c1a0dd0 con 0x7fd33c1041c0 2024-01-24T14:22:44.076 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:44.073+0000 7fd323fff700 1 -- 172.21.15.19:0/728629055 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3300173f0 con 0x7fd33c1041c0 2024-01-24T14:22:44.076 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:44.073+0000 7fd341766700 1 -- 172.21.15.19:0/728629055 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd33c1a1210 con 0x7fd33c1041c0 2024-01-24T14:22:44.077 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:44.074+0000 7fd341766700 1 -- 172.21.15.19:0/728629055 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd33c10c620 con 0x7fd33c1041c0 2024-01-24T14:22:44.077 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:44.075+0000 7fd323fff700 1 -- 172.21.15.19:0/728629055 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7fd33000fcc0 con 0x7fd33c1041c0 2024-01-24T14:22:44.077 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:44.076+0000 7fd323fff700 1 --2- 172.21.15.19:0/728629055 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fd3240a0520 0x7fd3240a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:22:44.077 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:44.076+0000 7fd33affd700 1 --2- 172.21.15.19:0/728629055 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fd3240a0520 0x7fd3240a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:22:44.077 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:44.076+0000 7fd323fff700 1 -- 172.21.15.19:0/728629055 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 4253+0+0 (secure 0 0 0) 0x7fd3300cbbb0 con 0x7fd33c1041c0 2024-01-24T14:22:44.077 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:44.076+0000 7fd33affd700 1 --2- 172.21.15.19:0/728629055 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fd3240a0520 0x7fd3240a29e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd32c005950 tx=0x7fd32c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:22:44.081 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:44.081+0000 7fd323fff700 1 -- 172.21.15.19:0/728629055 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fd330096720 con 0x7fd33c1041c0 2024-01-24T14:22:44.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:44.297+0000 7fd341766700 1 -- 172.21.15.19:0/728629055 --> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fd33c103450 con 0x7fd3240a0520 2024-01-24T14:22:44.710 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:44 smithi019 ceph-mon[40663]: Detected new or changed devices on smithi194 2024-01-24T14:22:44.710 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:44 smithi019 ceph-mon[40663]: Adjusting osd_memory_target on smithi194 to 8557M 2024-01-24T14:22:44.710 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:44 smithi019 ceph-mon[40663]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-01-24T14:22:44.710 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:44 smithi019 ceph-mon[40663]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-01-24T14:22:44.711 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:44 smithi019 ceph-mon[40663]: osdmap e34: 6 total, 5 up, 6 in 2024-01-24T14:22:44.711 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:44 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-24T14:22:44.711 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:44 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-24T14:22:44.711 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:44 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T14:22:44.711 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:44 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T14:22:44.712 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:44 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:22:44.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:44 smithi123 ceph-mon[44966]: Detected new or changed devices on smithi194 2024-01-24T14:22:44.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:44 smithi123 ceph-mon[44966]: Adjusting osd_memory_target on smithi194 to 8557M 2024-01-24T14:22:44.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:44 smithi123 ceph-mon[44966]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-01-24T14:22:44.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:44 smithi123 ceph-mon[44966]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-01-24T14:22:44.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:44 smithi123 ceph-mon[44966]: osdmap e34: 6 total, 5 up, 6 in 2024-01-24T14:22:44.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:44 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-24T14:22:44.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:44 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-24T14:22:44.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:44 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T14:22:44.801 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:44 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T14:22:44.801 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:44 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:22:44.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:44 smithi194 ceph-mon[45450]: Detected new or changed devices on smithi194 2024-01-24T14:22:44.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:44 smithi194 ceph-mon[45450]: Adjusting osd_memory_target on smithi194 to 8557M 2024-01-24T14:22:44.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:44 smithi194 ceph-mon[45450]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-01-24T14:22:44.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:44 smithi194 ceph-mon[45450]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-01-24T14:22:44.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:44 smithi194 ceph-mon[45450]: osdmap e34: 6 total, 5 up, 6 in 2024-01-24T14:22:44.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:44 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-24T14:22:44.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:44 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-24T14:22:44.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:44 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T14:22:44.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:44 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T14:22:44.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:44 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:22:45.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:45 smithi019 ceph-mon[40663]: purged_snaps scrub starts 2024-01-24T14:22:45.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:45 smithi019 ceph-mon[40663]: purged_snaps scrub ok 2024-01-24T14:22:45.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:45 smithi019 ceph-mon[40663]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:22:45.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:45 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-24T14:22:45.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:45 smithi019 ceph-mon[40663]: osd.5 [v2:172.21.15.194:6808/2483361871,v1:172.21.15.194:6809/2483361871] boot 2024-01-24T14:22:45.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:45 smithi019 ceph-mon[40663]: osdmap e35: 6 total, 6 up, 6 in 2024-01-24T14:22:45.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:45 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-24T14:22:45.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:45 smithi019 ceph-mon[40663]: osdmap e36: 6 total, 6 up, 6 in 2024-01-24T14:22:45.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:45 smithi123 ceph-mon[44966]: purged_snaps scrub starts 2024-01-24T14:22:45.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:45 smithi123 ceph-mon[44966]: purged_snaps scrub ok 2024-01-24T14:22:45.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:45 smithi123 ceph-mon[44966]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:22:45.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:45 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-24T14:22:45.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:45 smithi123 ceph-mon[44966]: osd.5 [v2:172.21.15.194:6808/2483361871,v1:172.21.15.194:6809/2483361871] boot 2024-01-24T14:22:45.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:45 smithi123 ceph-mon[44966]: osdmap e35: 6 total, 6 up, 6 in 2024-01-24T14:22:45.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:45 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-24T14:22:45.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:45 smithi123 ceph-mon[44966]: osdmap e36: 6 total, 6 up, 6 in 2024-01-24T14:22:45.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:45 smithi194 ceph-mon[45450]: purged_snaps scrub starts 2024-01-24T14:22:45.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:45 smithi194 ceph-mon[45450]: purged_snaps scrub ok 2024-01-24T14:22:45.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:45 smithi194 ceph-mon[45450]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:22:45.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:45 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-24T14:22:45.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:45 smithi194 ceph-mon[45450]: osd.5 [v2:172.21.15.194:6808/2483361871,v1:172.21.15.194:6809/2483361871] boot 2024-01-24T14:22:45.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:45 smithi194 ceph-mon[45450]: osdmap e35: 6 total, 6 up, 6 in 2024-01-24T14:22:45.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:45 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-01-24T14:22:45.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:45 smithi194 ceph-mon[45450]: osdmap e36: 6 total, 6 up, 6 in 2024-01-24T14:22:46.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:46 smithi019 ceph-mon[40663]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-01-24T14:22:46.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:46 smithi019 ceph-mon[40663]: osdmap e37: 6 total, 6 up, 6 in 2024-01-24T14:22:46.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:46 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/3080223243' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8c8ca2e-4cc6-4861-94b0-6211e4936c25"}]: dispatch 2024-01-24T14:22:46.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:46 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/3080223243' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c8c8ca2e-4cc6-4861-94b0-6211e4936c25"}]': finished 2024-01-24T14:22:46.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:46 smithi019 ceph-mon[40663]: osdmap e38: 7 total, 6 up, 7 in 2024-01-24T14:22:46.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:46 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-01-24T14:22:46.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:46 smithi123 ceph-mon[44966]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-01-24T14:22:46.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:46 smithi123 ceph-mon[44966]: osdmap e37: 6 total, 6 up, 6 in 2024-01-24T14:22:46.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:46 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/3080223243' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8c8ca2e-4cc6-4861-94b0-6211e4936c25"}]: dispatch 2024-01-24T14:22:46.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:46 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/3080223243' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c8c8ca2e-4cc6-4861-94b0-6211e4936c25"}]': finished 2024-01-24T14:22:46.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:46 smithi123 ceph-mon[44966]: osdmap e38: 7 total, 6 up, 7 in 2024-01-24T14:22:46.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:46 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-01-24T14:22:46.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:46 smithi194 ceph-mon[45450]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-01-24T14:22:46.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:46 smithi194 ceph-mon[45450]: osdmap e37: 6 total, 6 up, 6 in 2024-01-24T14:22:46.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:46 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/3080223243' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8c8ca2e-4cc6-4861-94b0-6211e4936c25"}]: dispatch 2024-01-24T14:22:46.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:46 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/3080223243' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c8c8ca2e-4cc6-4861-94b0-6211e4936c25"}]': finished 2024-01-24T14:22:46.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:46 smithi194 ceph-mon[45450]: osdmap e38: 7 total, 6 up, 7 in 2024-01-24T14:22:46.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:46 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-01-24T14:22:47.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:47 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/2396050624' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T14:22:47.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:47 smithi019 ceph-mon[40663]: osdmap e39: 7 total, 6 up, 7 in 2024-01-24T14:22:47.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:47 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-01-24T14:22:47.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:47 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/2396050624' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T14:22:47.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:47 smithi123 ceph-mon[44966]: osdmap e39: 7 total, 6 up, 7 in 2024-01-24T14:22:47.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:47 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-01-24T14:22:47.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:47 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/2396050624' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T14:22:47.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:47 smithi194 ceph-mon[45450]: osdmap e39: 7 total, 6 up, 7 in 2024-01-24T14:22:47.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:47 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-01-24T14:22:48.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:48 smithi019 ceph-mon[40663]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2024-01-24T14:22:48.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:48 smithi123 ceph-mon[44966]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2024-01-24T14:22:48.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:48 smithi194 ceph-mon[45450]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail 2024-01-24T14:22:50.702 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:50 smithi019 ceph-mon[40663]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail; 127 KiB/s, 0 objects/s recovering 2024-01-24T14:22:50.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:50 smithi123 ceph-mon[44966]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail; 127 KiB/s, 0 objects/s recovering 2024-01-24T14:22:50.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:50 smithi194 ceph-mon[45450]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 160 MiB used, 536 GiB / 536 GiB avail; 127 KiB/s, 0 objects/s recovering 2024-01-24T14:22:52.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:52 smithi019 ceph-mon[40663]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-01-24T14:22:52.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:52 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-01-24T14:22:52.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:52 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:22:52.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:52 smithi123 ceph-mon[44966]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-01-24T14:22:52.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:52 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-01-24T14:22:52.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:52 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:22:52.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:52 smithi194 ceph-mon[45450]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-01-24T14:22:52.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:52 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-01-24T14:22:52.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:52 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:22:53.725 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:53 smithi019 ceph-mon[40663]: Deploying daemon osd.6 on smithi019 2024-01-24T14:22:53.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:53 smithi123 ceph-mon[44966]: Deploying daemon osd.6 on smithi019 2024-01-24T14:22:53.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:53 smithi194 ceph-mon[45450]: Deploying daemon osd.6 on smithi019 2024-01-24T14:22:54.518 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:54 smithi019 ceph-mon[40663]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-01-24T14:22:54.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:54 smithi123 ceph-mon[44966]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-01-24T14:22:54.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:54 smithi194 ceph-mon[45450]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-01-24T14:22:56.441 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:56 smithi019 ceph-mon[40663]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-01-24T14:22:56.441 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:56 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:56.441 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:56 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:22:56.441 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:56 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:56.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:56 smithi123 ceph-mon[44966]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-01-24T14:22:56.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:56 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:56.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:56 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:22:56.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:56 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:56.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:56 smithi194 ceph-mon[45450]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-01-24T14:22:56.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:56 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:56.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:56 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:22:56.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:56 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:58.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:58.494+0000 7fd323fff700 1 -- 172.21.15.19:0/728629055 <== mgr.14152 v2:172.21.15.19:6800/1634350318 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd33c103450 con 0x7fd3240a0520 2024-01-24T14:22:58.496 INFO:teuthology.orchestra.run.smithi019.stdout:Created osd(s) 6 on host 'smithi019' 2024-01-24T14:22:58.498 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:58.497+0000 7fd341766700 1 -- 172.21.15.19:0/728629055 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fd3240a0520 msgr2=0x7fd3240a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:22:58.498 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:58.497+0000 7fd341766700 1 --2- 172.21.15.19:0/728629055 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fd3240a0520 0x7fd3240a29e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd32c005950 tx=0x7fd32c00b410 comp rx=0 tx=0).stop 2024-01-24T14:22:58.498 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:58.497+0000 7fd341766700 1 -- 172.21.15.19:0/728629055 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd33c1041c0 msgr2=0x7fd33c0ff840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:22:58.498 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:58.497+0000 7fd341766700 1 --2- 172.21.15.19:0/728629055 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd33c1041c0 0x7fd33c0ff840 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd330000c00 tx=0x7fd33000f710 comp rx=0 tx=0).stop 2024-01-24T14:22:58.498 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:58.497+0000 7fd341766700 1 -- 172.21.15.19:0/728629055 shutdown_connections 2024-01-24T14:22:58.498 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:58.497+0000 7fd341766700 1 --2- 172.21.15.19:0/728629055 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd33c106af0 0x7fd33c0ffd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:22:58.498 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:58.497+0000 7fd341766700 1 --2- 172.21.15.19:0/728629055 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd33c101890 0x7fd33c1011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:22:58.499 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:58.497+0000 7fd341766700 1 --2- 172.21.15.19:0/728629055 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fd3240a0520 0x7fd3240a29e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:22:58.499 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:58.497+0000 7fd341766700 1 --2- 172.21.15.19:0/728629055 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd33c1041c0 0x7fd33c0ff840 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:22:58.499 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:58.497+0000 7fd341766700 1 -- 172.21.15.19:0/728629055 >> 172.21.15.19:0/728629055 conn(0x7fd33c0fb2c0 msgr2=0x7fd33c102d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:22:58.499 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:58.497+0000 7fd341766700 1 -- 172.21.15.19:0/728629055 shutdown_connections 2024-01-24T14:22:58.499 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:22:58.497+0000 7fd341766700 1 -- 172.21.15.19:0/728629055 wait complete. 2024-01-24T14:22:58.667 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:58 smithi019 ceph-mon[40663]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-01-24T14:22:58.667 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:58 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:58.667 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:58 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:58.667 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:58 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:22:58.667 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:58 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:22:58.667 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:58 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:58.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:58 smithi123 ceph-mon[44966]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-01-24T14:22:58.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:58 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:58.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:58 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:58.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:58 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:22:58.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:58 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:22:58.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:58 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:58.842 DEBUG:teuthology.orchestra.run.smithi019:osd.6> sudo journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@osd.6.service 2024-01-24T14:22:58.843 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:58 smithi194 ceph-mon[45450]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-01-24T14:22:58.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:58 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:58.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:58 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:58.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:58 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:22:58.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:58 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:22:58.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:58 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:58.845 INFO:tasks.cephadm:Deploying osd.7 on smithi123 with /dev/vg_nvme/lv_2... 2024-01-24T14:22:58.845 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-01-24T14:22:58.883 INFO:journalctl@ceph.osd.6.smithi019.stdout:-- Logs begin at Wed 2024-01-24 14:03:52 UTC. -- 2024-01-24T14:22:59.077 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/mon.b/config 2024-01-24T14:22:59.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:59 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:59.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:59 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:59.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:59 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:22:59.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:22:59 smithi019 ceph-mon[40663]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-01-24T14:22:59.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:59 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:59.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:59 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:59.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:59 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:22:59.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:22:59 smithi123 ceph-mon[44966]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-01-24T14:22:59.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:59 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:59.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:59 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:22:59.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:59 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:22:59.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:22:59 smithi194 ceph-mon[45450]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-01-24T14:23:00.289 INFO:teuthology.orchestra.run.smithi123.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-01-24T14:23:00.289 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-01-24T14:23:00.289 INFO:teuthology.orchestra.run.smithi123.stderr: stderr: 10+0 records in 2024-01-24T14:23:00.289 INFO:teuthology.orchestra.run.smithi123.stderr:10+0 records out 2024-01-24T14:23:00.289 INFO:teuthology.orchestra.run.smithi123.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113313 s, 925 MB/s 2024-01-24T14:23:00.289 INFO:teuthology.orchestra.run.smithi123.stderr:--> Zapping successful for: 2024-01-24T14:23:00.467 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:23:00 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6[76344]: 2024-01-24T14:23:00.074+0000 7f6f47668640 -1 osd.6 0 log_to_monitors true 2024-01-24T14:23:00.615 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:00 smithi123 ceph-mon[44966]: from='osd.6 [v2:172.21.15.19:6818/1290513386,v1:172.21.15.19:6819/1290513386]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-01-24T14:23:00.709 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 -- ceph orch daemon add osd smithi123:vg_nvme/lv_2 2024-01-24T14:23:00.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:00 smithi019 ceph-mon[40663]: from='osd.6 [v2:172.21.15.19:6818/1290513386,v1:172.21.15.19:6819/1290513386]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-01-24T14:23:00.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:00 smithi194 ceph-mon[45450]: from='osd.6 [v2:172.21.15.19:6818/1290513386,v1:172.21.15.19:6819/1290513386]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-01-24T14:23:00.925 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/mon.b/config 2024-01-24T14:23:01.481 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:01 smithi019 ceph-mon[40663]: from='osd.6 [v2:172.21.15.19:6818/1290513386,v1:172.21.15.19:6819/1290513386]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-01-24T14:23:01.481 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:01 smithi019 ceph-mon[40663]: osdmap e40: 7 total, 6 up, 7 in 2024-01-24T14:23:01.481 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:01 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-01-24T14:23:01.736 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:23:01 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6[76344]: 2024-01-24T14:23:01.488+0000 7f6f3d4ce700 -1 osd.6 0 waiting for initial osdmap 2024-01-24T14:23:01.736 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:23:01 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6[76344]: 2024-01-24T14:23:01.499+0000 7f6f34aba700 -1 osd.6 41 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-24T14:23:01.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:01 smithi019 ceph-mon[40663]: from='osd.6 [v2:172.21.15.19:6818/1290513386,v1:172.21.15.19:6819/1290513386]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-01-24T14:23:01.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:01 smithi019 ceph-mon[40663]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-01-24T14:23:01.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:01 smithi019 ceph-mon[40663]: Detected new or changed devices on smithi019 2024-01-24T14:23:01.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:01 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:01.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:01 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:01.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:01 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:23:01.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:01 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:23:01.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:01 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:23:01.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:01 smithi019 ceph-mon[40663]: Adjusting osd_memory_target on smithi019 to 5706M 2024-01-24T14:23:01.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:01 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:01.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:01 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:23:01.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:01 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:23:01.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:01 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:01.754 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:01 smithi123 ceph-mon[44966]: from='osd.6 [v2:172.21.15.19:6818/1290513386,v1:172.21.15.19:6819/1290513386]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-01-24T14:23:01.754 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:01 smithi123 ceph-mon[44966]: osdmap e40: 7 total, 6 up, 7 in 2024-01-24T14:23:01.754 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:01 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-01-24T14:23:01.754 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:01 smithi123 ceph-mon[44966]: from='osd.6 [v2:172.21.15.19:6818/1290513386,v1:172.21.15.19:6819/1290513386]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-01-24T14:23:01.755 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:01 smithi123 ceph-mon[44966]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-01-24T14:23:01.755 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:01 smithi123 ceph-mon[44966]: Detected new or changed devices on smithi019 2024-01-24T14:23:01.755 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:01 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:01.755 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:01 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:01.755 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:01 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:23:01.755 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:01 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:23:01.755 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:01 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:23:01.755 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:01 smithi123 ceph-mon[44966]: Adjusting osd_memory_target on smithi019 to 5706M 2024-01-24T14:23:01.755 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:01 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:01.755 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:01 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:23:01.755 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:01 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:23:01.755 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:01 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:01.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:01 smithi194 ceph-mon[45450]: from='osd.6 [v2:172.21.15.19:6818/1290513386,v1:172.21.15.19:6819/1290513386]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-01-24T14:23:01.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:01 smithi194 ceph-mon[45450]: osdmap e40: 7 total, 6 up, 7 in 2024-01-24T14:23:01.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:01 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-01-24T14:23:01.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:01 smithi194 ceph-mon[45450]: from='osd.6 [v2:172.21.15.19:6818/1290513386,v1:172.21.15.19:6819/1290513386]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-01-24T14:23:01.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:01 smithi194 ceph-mon[45450]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-01-24T14:23:01.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:01 smithi194 ceph-mon[45450]: Detected new or changed devices on smithi019 2024-01-24T14:23:01.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:01 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:01.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:01 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:01.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:01 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:23:01.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:01 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:23:01.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:01 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:23:01.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:01 smithi194 ceph-mon[45450]: Adjusting osd_memory_target on smithi019 to 5706M 2024-01-24T14:23:01.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:01 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:01.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:01 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:23:01.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:01 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:23:01.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:01 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:01.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:01.922+0000 7f83c8f8b700 1 -- 172.21.15.123:0/1924748737 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83c4100ed0 msgr2=0x7f83c4101330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:23:01.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:01.922+0000 7f83c8f8b700 1 --2- 172.21.15.123:0/1924748737 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83c4100ed0 0x7f83c4101330 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f83b40099c0 tx=0x7f83b4009cd0 comp rx=0 tx=0).stop 2024-01-24T14:23:01.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:01.922+0000 7f83c8f8b700 1 -- 172.21.15.123:0/1924748737 shutdown_connections 2024-01-24T14:23:01.927 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:01.922+0000 7f83c8f8b700 1 --2- 172.21.15.123:0/1924748737 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f83c4101870 0x7f83c4108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:23:01.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:01.922+0000 7f83c8f8b700 1 --2- 172.21.15.123:0/1924748737 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83c40ffcd0 0x7f83c41000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:23:01.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:01.922+0000 7f83c8f8b700 1 --2- 172.21.15.123:0/1924748737 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83c4100ed0 0x7f83c4101330 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:23:01.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:01.922+0000 7f83c8f8b700 1 -- 172.21.15.123:0/1924748737 >> 172.21.15.123:0/1924748737 conn(0x7f83c40fb250 msgr2=0x7f83c40fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:23:01.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:01.922+0000 7f83c8f8b700 1 -- 172.21.15.123:0/1924748737 shutdown_connections 2024-01-24T14:23:01.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:01.922+0000 7f83c8f8b700 1 -- 172.21.15.123:0/1924748737 wait complete. 2024-01-24T14:23:01.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:01.923+0000 7f83c8f8b700 1 Processor -- start 2024-01-24T14:23:01.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:01.923+0000 7f83c8f8b700 1 -- start start 2024-01-24T14:23:01.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:01.923+0000 7f83c8f8b700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f83c40ffcd0 0x7f83c4196b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:23:01.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:01.923+0000 7f83c8f8b700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83c4100ed0 0x7f83c41970b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:23:01.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:01.923+0000 7f83c8f8b700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83c4101870 0x7f83c419e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:23:01.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:01.923+0000 7f83c8f8b700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c419e700 con 0x7f83c40ffcd0 2024-01-24T14:23:01.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:01.923+0000 7f83c8f8b700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c419e840 con 0x7f83c4101870 2024-01-24T14:23:01.928 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:01.923+0000 7f83c8f8b700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c419e9d0 con 0x7f83c4100ed0 2024-01-24T14:23:01.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:01.923+0000 7f83c2d9d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83c4101870 0x7f83c419e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:23:01.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:01.923+0000 7f83c2d9d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83c4101870 0x7f83c419e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:33416/0 (socket says 172.21.15.123:33416) 2024-01-24T14:23:01.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:01.923+0000 7f83c2d9d700 1 -- 172.21.15.123:0/4266821864 learned_addr learned my addr 172.21.15.123:0/4266821864 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-24T14:23:01.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:01.923+0000 7f83c1d9b700 1 --2- 172.21.15.123:0/4266821864 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83c4100ed0 0x7f83c41970b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:23:01.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:01.923+0000 7f83c259c700 1 --2- 172.21.15.123:0/4266821864 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f83c40ffcd0 0x7f83c4196b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:23:01.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:01.923+0000 7f83c2d9d700 1 -- 172.21.15.123:0/4266821864 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83c4100ed0 msgr2=0x7f83c41970b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:23:01.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:01.923+0000 7f83c2d9d700 1 --2- 172.21.15.123:0/4266821864 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83c4100ed0 0x7f83c41970b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:23:01.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:01.923+0000 7f83c2d9d700 1 -- 172.21.15.123:0/4266821864 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f83c40ffcd0 msgr2=0x7f83c4196b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:23:01.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:01.923+0000 7f83c2d9d700 1 --2- 172.21.15.123:0/4266821864 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f83c40ffcd0 0x7f83c4196b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:23:01.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:01.923+0000 7f83c2d9d700 1 -- 172.21.15.123:0/4266821864 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b40095f0 con 0x7f83c4101870 2024-01-24T14:23:01.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:01.923+0000 7f83c259c700 1 --2- 172.21.15.123:0/4266821864 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f83c40ffcd0 0x7f83c4196b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T14:23:01.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:01.924+0000 7f83c2d9d700 1 --2- 172.21.15.123:0/4266821864 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83c4101870 0x7f83c419e130 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f83bc00b970 tx=0x7f83bc00bd30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:23:01.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:01.924+0000 7f83b37fe700 1 -- 172.21.15.123:0/4266821864 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83bc00c9b0 con 0x7f83c4101870 2024-01-24T14:23:01.929 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:01.924+0000 7f83c8f8b700 1 -- 172.21.15.123:0/4266821864 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83c419ecb0 con 0x7f83c4101870 2024-01-24T14:23:01.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:01.924+0000 7f83b37fe700 1 -- 172.21.15.123:0/4266821864 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f83bc014440 con 0x7f83c4101870 2024-01-24T14:23:01.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:01.924+0000 7f83b37fe700 1 -- 172.21.15.123:0/4266821864 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83bc0075d0 con 0x7f83c4101870 2024-01-24T14:23:01.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:01.924+0000 7f83c8f8b700 1 -- 172.21.15.123:0/4266821864 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83c419f160 con 0x7f83c4101870 2024-01-24T14:23:01.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:01.925+0000 7f83c8f8b700 1 -- 172.21.15.123:0/4266821864 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83c410b5f0 con 0x7f83c4101870 2024-01-24T14:23:01.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:01.926+0000 7f83b37fe700 1 -- 172.21.15.123:0/4266821864 <== mon.2 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f83bc0148d0 con 0x7f83c4101870 2024-01-24T14:23:01.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:01.926+0000 7f83b37fe700 1 --2- 172.21.15.123:0/4266821864 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f83ac0a0520 0x7f83ac0a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:23:01.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:01.927+0000 7f83b37fe700 1 -- 172.21.15.123:0/4266821864 <== mon.2 v2:172.21.15.123:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4685+0+0 (secure 0 0 0) 0x7f83bc006e40 con 0x7f83c4101870 2024-01-24T14:23:01.930 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:01.927+0000 7f83c259c700 1 --2- 172.21.15.123:0/4266821864 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f83ac0a0520 0x7f83ac0a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:23:01.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:01.927+0000 7f83c259c700 1 --2- 172.21.15.123:0/4266821864 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f83ac0a0520 0x7f83ac0a29e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f83c4100d30 tx=0x7f83b8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:23:01.931 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:01.930+0000 7f83b37fe700 1 -- 172.21.15.123:0/4266821864 <== mon.2 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f83bc0950b0 con 0x7f83c4101870 2024-01-24T14:23:02.126 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:02.126+0000 7f83c8f8b700 1 -- 172.21.15.123:0/4266821864 --> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f83c40697e0 con 0x7f83ac0a0520 2024-01-24T14:23:02.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:02 smithi123 ceph-mon[44966]: from='osd.6 [v2:172.21.15.19:6818/1290513386,v1:172.21.15.19:6819/1290513386]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]': finished 2024-01-24T14:23:02.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:02 smithi123 ceph-mon[44966]: osdmap e41: 7 total, 6 up, 7 in 2024-01-24T14:23:02.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:02 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-01-24T14:23:02.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:02 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-01-24T14:23:02.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:02 smithi123 ceph-mon[44966]: from='client.24287 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:23:02.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:02 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T14:23:02.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:02 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T14:23:02.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:02 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:23:02.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:02 smithi194 ceph-mon[45450]: from='osd.6 [v2:172.21.15.19:6818/1290513386,v1:172.21.15.19:6819/1290513386]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]': finished 2024-01-24T14:23:02.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:02 smithi194 ceph-mon[45450]: osdmap e41: 7 total, 6 up, 7 in 2024-01-24T14:23:02.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:02 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-01-24T14:23:02.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:02 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-01-24T14:23:02.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:02 smithi194 ceph-mon[45450]: from='client.24287 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:23:02.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:02 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T14:23:02.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:02 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T14:23:02.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:02 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:23:02.985 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:02 smithi019 ceph-mon[40663]: from='osd.6 [v2:172.21.15.19:6818/1290513386,v1:172.21.15.19:6819/1290513386]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]': finished 2024-01-24T14:23:02.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:02 smithi019 ceph-mon[40663]: osdmap e41: 7 total, 6 up, 7 in 2024-01-24T14:23:02.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:02 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-01-24T14:23:02.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:02 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-01-24T14:23:02.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:02 smithi019 ceph-mon[40663]: from='client.24287 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:23:02.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:02 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T14:23:02.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:02 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T14:23:02.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:02 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:23:03.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:03 smithi123 ceph-mon[44966]: purged_snaps scrub starts 2024-01-24T14:23:03.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:03 smithi123 ceph-mon[44966]: purged_snaps scrub ok 2024-01-24T14:23:03.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:03 smithi123 ceph-mon[44966]: osd.6 [v2:172.21.15.19:6818/1290513386,v1:172.21.15.19:6819/1290513386] boot 2024-01-24T14:23:03.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:03 smithi123 ceph-mon[44966]: osdmap e42: 7 total, 7 up, 7 in 2024-01-24T14:23:03.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:03 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-01-24T14:23:03.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:03 smithi123 ceph-mon[44966]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 626 GiB / 626 GiB avail 2024-01-24T14:23:03.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:03 smithi194 ceph-mon[45450]: purged_snaps scrub starts 2024-01-24T14:23:03.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:03 smithi194 ceph-mon[45450]: purged_snaps scrub ok 2024-01-24T14:23:03.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:03 smithi194 ceph-mon[45450]: osd.6 [v2:172.21.15.19:6818/1290513386,v1:172.21.15.19:6819/1290513386] boot 2024-01-24T14:23:03.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:03 smithi194 ceph-mon[45450]: osdmap e42: 7 total, 7 up, 7 in 2024-01-24T14:23:03.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:03 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-01-24T14:23:03.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:03 smithi194 ceph-mon[45450]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 626 GiB / 626 GiB avail 2024-01-24T14:23:03.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:03 smithi019 ceph-mon[40663]: purged_snaps scrub starts 2024-01-24T14:23:03.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:03 smithi019 ceph-mon[40663]: purged_snaps scrub ok 2024-01-24T14:23:03.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:03 smithi019 ceph-mon[40663]: osd.6 [v2:172.21.15.19:6818/1290513386,v1:172.21.15.19:6819/1290513386] boot 2024-01-24T14:23:03.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:03 smithi019 ceph-mon[40663]: osdmap e42: 7 total, 7 up, 7 in 2024-01-24T14:23:03.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:03 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-01-24T14:23:03.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:03 smithi019 ceph-mon[40663]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 626 GiB / 626 GiB avail 2024-01-24T14:23:04.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:04 smithi123 ceph-mon[44966]: osdmap e43: 7 total, 7 up, 7 in 2024-01-24T14:23:04.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:04 smithi123 ceph-mon[44966]: from='client.? 172.21.15.123:0/2834945025' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9aaea7a8-7bc3-4e32-9328-20a6c30a6d8f"}]: dispatch 2024-01-24T14:23:04.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:04 smithi123 ceph-mon[44966]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9aaea7a8-7bc3-4e32-9328-20a6c30a6d8f"}]: dispatch 2024-01-24T14:23:04.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:04 smithi123 ceph-mon[44966]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9aaea7a8-7bc3-4e32-9328-20a6c30a6d8f"}]': finished 2024-01-24T14:23:04.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:04 smithi123 ceph-mon[44966]: osdmap e44: 8 total, 7 up, 8 in 2024-01-24T14:23:04.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:04 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-01-24T14:23:04.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:04 smithi194 ceph-mon[45450]: osdmap e43: 7 total, 7 up, 7 in 2024-01-24T14:23:04.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:04 smithi194 ceph-mon[45450]: from='client.? 172.21.15.123:0/2834945025' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9aaea7a8-7bc3-4e32-9328-20a6c30a6d8f"}]: dispatch 2024-01-24T14:23:04.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:04 smithi194 ceph-mon[45450]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9aaea7a8-7bc3-4e32-9328-20a6c30a6d8f"}]: dispatch 2024-01-24T14:23:04.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:04 smithi194 ceph-mon[45450]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9aaea7a8-7bc3-4e32-9328-20a6c30a6d8f"}]': finished 2024-01-24T14:23:04.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:04 smithi194 ceph-mon[45450]: osdmap e44: 8 total, 7 up, 8 in 2024-01-24T14:23:04.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:04 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-01-24T14:23:04.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:04 smithi019 ceph-mon[40663]: osdmap e43: 7 total, 7 up, 7 in 2024-01-24T14:23:04.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:04 smithi019 ceph-mon[40663]: from='client.? 172.21.15.123:0/2834945025' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9aaea7a8-7bc3-4e32-9328-20a6c30a6d8f"}]: dispatch 2024-01-24T14:23:04.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:04 smithi019 ceph-mon[40663]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9aaea7a8-7bc3-4e32-9328-20a6c30a6d8f"}]: dispatch 2024-01-24T14:23:04.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:04 smithi019 ceph-mon[40663]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9aaea7a8-7bc3-4e32-9328-20a6c30a6d8f"}]': finished 2024-01-24T14:23:04.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:04 smithi019 ceph-mon[40663]: osdmap e44: 8 total, 7 up, 8 in 2024-01-24T14:23:04.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:04 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-01-24T14:23:05.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:05 smithi123 ceph-mon[44966]: from='client.? 172.21.15.123:0/3523150793' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T14:23:05.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:05 smithi123 ceph-mon[44966]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 626 GiB / 626 GiB avail 2024-01-24T14:23:05.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:05 smithi194 ceph-mon[45450]: from='client.? 172.21.15.123:0/3523150793' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T14:23:05.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:05 smithi194 ceph-mon[45450]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 626 GiB / 626 GiB avail 2024-01-24T14:23:05.985 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:05 smithi019 ceph-mon[40663]: from='client.? 172.21.15.123:0/3523150793' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T14:23:05.985 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:05 smithi019 ceph-mon[40663]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 626 GiB / 626 GiB avail 2024-01-24T14:23:08.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:08 smithi123 ceph-mon[44966]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T14:23:08.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:08 smithi194 ceph-mon[45450]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T14:23:08.485 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:08 smithi019 ceph-mon[40663]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T14:23:10.451 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:10 smithi123 ceph-mon[44966]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T14:23:10.452 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:10 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-01-24T14:23:10.452 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:10 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:23:10.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:10 smithi019 ceph-mon[40663]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T14:23:10.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:10 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-01-24T14:23:10.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:10 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:23:10.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:10 smithi194 ceph-mon[45450]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T14:23:10.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:10 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-01-24T14:23:10.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:10 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:23:11.505 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:11 smithi123 ceph-mon[44966]: Deploying daemon osd.7 on smithi123 2024-01-24T14:23:11.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:11 smithi019 ceph-mon[40663]: Deploying daemon osd.7 on smithi123 2024-01-24T14:23:11.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:11 smithi194 ceph-mon[45450]: Deploying daemon osd.7 on smithi123 2024-01-24T14:23:12.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:12 smithi123 ceph-mon[44966]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T14:23:12.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:12 smithi019 ceph-mon[40663]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T14:23:12.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:12 smithi194 ceph-mon[45450]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T14:23:14.521 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:14 smithi123 ceph-mon[44966]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T14:23:14.521 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:14 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:14.521 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:14 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:14.521 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:14 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:23:14.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:14 smithi019 ceph-mon[40663]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T14:23:14.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:14 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:14.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:14 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:14.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:14 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:23:14.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:14 smithi194 ceph-mon[45450]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T14:23:14.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:14 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:14.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:14 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:14.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:14 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:23:15.680 INFO:teuthology.orchestra.run.smithi123.stdout:Created osd(s) 7 on host 'smithi123' 2024-01-24T14:23:15.681 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:15.679+0000 7f83b37fe700 1 -- 172.21.15.123:0/4266821864 <== mgr.14152 v2:172.21.15.19:6800/1634350318 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f83c40697e0 con 0x7f83ac0a0520 2024-01-24T14:23:15.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:15.682+0000 7f83c8f8b700 1 -- 172.21.15.123:0/4266821864 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f83ac0a0520 msgr2=0x7f83ac0a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:23:15.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:15.682+0000 7f83c8f8b700 1 --2- 172.21.15.123:0/4266821864 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f83ac0a0520 0x7f83ac0a29e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f83c4100d30 tx=0x7f83b8008040 comp rx=0 tx=0).stop 2024-01-24T14:23:15.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:15.682+0000 7f83c8f8b700 1 -- 172.21.15.123:0/4266821864 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83c4101870 msgr2=0x7f83c419e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:23:15.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:15.682+0000 7f83c8f8b700 1 --2- 172.21.15.123:0/4266821864 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83c4101870 0x7f83c419e130 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f83bc00b970 tx=0x7f83bc00bd30 comp rx=0 tx=0).stop 2024-01-24T14:23:15.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:15.682+0000 7f83c8f8b700 1 -- 172.21.15.123:0/4266821864 shutdown_connections 2024-01-24T14:23:15.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:15.682+0000 7f83c8f8b700 1 --2- 172.21.15.123:0/4266821864 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f83ac0a0520 0x7f83ac0a29e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:23:15.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:15.682+0000 7f83c8f8b700 1 --2- 172.21.15.123:0/4266821864 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f83c40ffcd0 0x7f83c4196b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:23:15.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:15.682+0000 7f83c8f8b700 1 --2- 172.21.15.123:0/4266821864 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83c4101870 0x7f83c419e130 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:23:15.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:15.682+0000 7f83c8f8b700 1 --2- 172.21.15.123:0/4266821864 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83c4100ed0 0x7f83c41970b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:23:15.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:15.682+0000 7f83c8f8b700 1 -- 172.21.15.123:0/4266821864 >> 172.21.15.123:0/4266821864 conn(0x7f83c40fb250 msgr2=0x7f83c40690d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:23:15.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:15.682+0000 7f83c8f8b700 1 -- 172.21.15.123:0/4266821864 shutdown_connections 2024-01-24T14:23:15.683 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:15.682+0000 7f83c8f8b700 1 -- 172.21.15.123:0/4266821864 wait complete. 2024-01-24T14:23:16.143 DEBUG:teuthology.orchestra.run.smithi123:osd.7> sudo journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@osd.7.service 2024-01-24T14:23:16.145 INFO:tasks.cephadm:Deploying osd.8 on smithi194 with /dev/vg_nvme/lv_2... 2024-01-24T14:23:16.145 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-01-24T14:23:16.182 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:16 smithi123 ceph-mon[44966]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T14:23:16.182 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:16 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:16.182 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:16 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:16.182 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:16 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:23:16.182 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:16 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:23:16.182 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:16 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:16.182 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:16 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:16.183 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:16 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:16.183 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:16 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:23:16.188 INFO:journalctl@ceph.osd.7.smithi123.stdout:-- Logs begin at Wed 2024-01-24 14:03:45 UTC. -- 2024-01-24T14:23:16.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:16 smithi194 ceph-mon[45450]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T14:23:16.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:16 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:16.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:16 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:16.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:16 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:23:16.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:16 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:23:16.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:16 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:16.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:16 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:16.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:16 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:16.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:16 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:23:16.392 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/mon.c/config 2024-01-24T14:23:16.485 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:16 smithi019 ceph-mon[40663]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T14:23:16.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:16 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:16.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:16 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:16.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:16 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:23:16.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:16 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:23:16.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:16 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:16.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:16 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:16.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:16 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:16.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:16 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:23:17.617 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-01-24T14:23:17.617 INFO:teuthology.orchestra.run.smithi194.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-01-24T14:23:17.617 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10+0 records in 2024-01-24T14:23:17.617 INFO:teuthology.orchestra.run.smithi194.stderr:10+0 records out 2024-01-24T14:23:17.617 INFO:teuthology.orchestra.run.smithi194.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.011254 s, 932 MB/s 2024-01-24T14:23:17.617 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping successful for: 2024-01-24T14:23:17.800 INFO:journalctl@ceph.osd.7.smithi123.stdout:Jan 24 14:23:17 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-7[66344]: 2024-01-24T14:23:17.542+0000 7fd8df786640 -1 osd.7 0 log_to_monitors true 2024-01-24T14:23:18.025 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 -- ceph orch daemon add osd smithi194:vg_nvme/lv_2 2024-01-24T14:23:18.154 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:18 smithi123 ceph-mon[44966]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T14:23:18.155 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:18 smithi123 ceph-mon[44966]: from='osd.7 [v2:172.21.15.123:6816/1391966446,v1:172.21.15.123:6817/1391966446]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-01-24T14:23:18.155 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:18 smithi123 ceph-mon[44966]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-01-24T14:23:18.247 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/mon.c/config 2024-01-24T14:23:18.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:18 smithi194 ceph-mon[45450]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T14:23:18.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:18 smithi194 ceph-mon[45450]: from='osd.7 [v2:172.21.15.123:6816/1391966446,v1:172.21.15.123:6817/1391966446]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-01-24T14:23:18.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:18 smithi194 ceph-mon[45450]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-01-24T14:23:18.485 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:18 smithi019 ceph-mon[40663]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T14:23:18.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:18 smithi019 ceph-mon[40663]: from='osd.7 [v2:172.21.15.123:6816/1391966446,v1:172.21.15.123:6817/1391966446]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-01-24T14:23:18.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:18 smithi019 ceph-mon[40663]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-01-24T14:23:19.182 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:19.180+0000 7f27f8498700 1 -- 172.21.15.194:0/3674403947 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f27f0102920 msgr2=0x7f27f0109240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:23:19.182 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:19.180+0000 7f27f8498700 1 --2- 172.21.15.194:0/3674403947 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f27f0102920 0x7f27f0109240 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f27e800b0a0 tx=0x7f27e800b3b0 comp rx=0 tx=0).stop 2024-01-24T14:23:19.183 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:19.180+0000 7f27f8498700 1 -- 172.21.15.194:0/3674403947 shutdown_connections 2024-01-24T14:23:19.183 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:19.180+0000 7f27f8498700 1 --2- 172.21.15.194:0/3674403947 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f27f0100e00 0x7f27f0101220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:23:19.183 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:19.180+0000 7f27f8498700 1 --2- 172.21.15.194:0/3674403947 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f27f0102920 0x7f27f0109240 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:23:19.183 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:19.180+0000 7f27f8498700 1 --2- 172.21.15.194:0/3674403947 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27f0101f60 0x7f27f01023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:23:19.183 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:19.180+0000 7f27f8498700 1 -- 172.21.15.194:0/3674403947 >> 172.21.15.194:0/3674403947 conn(0x7f27f00fc360 msgr2=0x7f27f00fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:23:19.183 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:19.180+0000 7f27f8498700 1 -- 172.21.15.194:0/3674403947 shutdown_connections 2024-01-24T14:23:19.183 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:19.180+0000 7f27f8498700 1 -- 172.21.15.194:0/3674403947 wait complete. 2024-01-24T14:23:19.183 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:19.181+0000 7f27f8498700 1 Processor -- start 2024-01-24T14:23:19.183 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:19.181+0000 7f27f8498700 1 -- start start 2024-01-24T14:23:19.183 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:19.181+0000 7f27f8498700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f27f0100e00 0x7f27f0074ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:23:19.183 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:19.181+0000 7f27f8498700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27f0101f60 0x7f27f0073130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:23:19.183 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:19.181+0000 7f27f8498700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f27f0102920 0x7f27f0073670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:23:19.183 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:19.181+0000 7f27f8498700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f0073df0 con 0x7f27f0102920 2024-01-24T14:23:19.183 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:19.181+0000 7f27f8498700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f0073f60 con 0x7f27f0101f60 2024-01-24T14:23:19.184 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:19.181+0000 7f27f8498700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f01a0b40 con 0x7f27f0100e00 2024-01-24T14:23:19.184 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:19.181+0000 7f27f6234700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f27f0100e00 0x7f27f0074ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:23:19.184 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:19.181+0000 7f27f6234700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f27f0100e00 0x7f27f0074ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:50532/0 (socket says 172.21.15.194:50532) 2024-01-24T14:23:19.184 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:19.181+0000 7f27f6234700 1 -- 172.21.15.194:0/2058147343 learned_addr learned my addr 172.21.15.194:0/2058147343 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-24T14:23:19.184 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:19.181+0000 7f27f6a35700 1 --2- 172.21.15.194:0/2058147343 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f27f0102920 0x7f27f0073670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:23:19.184 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:19.181+0000 7f27f5a33700 1 --2- 172.21.15.194:0/2058147343 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27f0101f60 0x7f27f0073130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:23:19.184 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:19.181+0000 7f27f6234700 1 -- 172.21.15.194:0/2058147343 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27f0101f60 msgr2=0x7f27f0073130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:23:19.184 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:19.181+0000 7f27f6234700 1 --2- 172.21.15.194:0/2058147343 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27f0101f60 0x7f27f0073130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:23:19.184 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:19.181+0000 7f27f6234700 1 -- 172.21.15.194:0/2058147343 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f27f0102920 msgr2=0x7f27f0073670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:23:19.184 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:19.181+0000 7f27f6234700 1 --2- 172.21.15.194:0/2058147343 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f27f0102920 0x7f27f0073670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:23:19.184 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:19.181+0000 7f27f6234700 1 -- 172.21.15.194:0/2058147343 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e8009d00 con 0x7f27f0100e00 2024-01-24T14:23:19.184 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:19.181+0000 7f27f6a35700 1 --2- 172.21.15.194:0/2058147343 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f27f0102920 0x7f27f0073670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T14:23:19.184 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:19.181+0000 7f27f5a33700 1 --2- 172.21.15.194:0/2058147343 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27f0101f60 0x7f27f0073130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T14:23:19.184 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:19.181+0000 7f27f6234700 1 --2- 172.21.15.194:0/2058147343 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f27f0100e00 0x7f27f0074ae0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f27e000b530 tx=0x7f27e000b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:23:19.185 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:19.181+0000 7f27e77fe700 1 -- 172.21.15.194:0/2058147343 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e0011830 con 0x7f27f0100e00 2024-01-24T14:23:19.185 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:19.182+0000 7f27e77fe700 1 -- 172.21.15.194:0/2058147343 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f27e0011e50 con 0x7f27f0100e00 2024-01-24T14:23:19.185 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:19.182+0000 7f27f8498700 1 -- 172.21.15.194:0/2058147343 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27f01a0e20 con 0x7f27f0100e00 2024-01-24T14:23:19.185 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:19.182+0000 7f27e77fe700 1 -- 172.21.15.194:0/2058147343 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e000f2e0 con 0x7f27f0100e00 2024-01-24T14:23:19.185 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:19.182+0000 7f27f8498700 1 -- 172.21.15.194:0/2058147343 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27f01a12b0 con 0x7f27f0100e00 2024-01-24T14:23:19.185 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:19.182+0000 7f27f8498700 1 -- 172.21.15.194:0/2058147343 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27f01083f0 con 0x7f27f0100e00 2024-01-24T14:23:19.186 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:19.183+0000 7f27e77fe700 1 -- 172.21.15.194:0/2058147343 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f27e0010580 con 0x7f27f0100e00 2024-01-24T14:23:19.186 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:19.184+0000 7f27e77fe700 1 --2- 172.21.15.194:0/2058147343 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f27dc0a0520 0x7f27dc0a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:23:19.186 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:19.184+0000 7f27e77fe700 1 -- 172.21.15.194:0/2058147343 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5117+0+0 (secure 0 0 0) 0x7f27e00ca830 con 0x7f27f0100e00 2024-01-24T14:23:19.186 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:19.184+0000 7f27f5a33700 1 --2- 172.21.15.194:0/2058147343 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f27dc0a0520 0x7f27dc0a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:23:19.186 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:19.185+0000 7f27f5a33700 1 --2- 172.21.15.194:0/2058147343 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f27dc0a0520 0x7f27dc0a29e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f27ec007900 tx=0x7f27ec008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:23:19.188 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:19.188+0000 7f27e77fe700 1 -- 172.21.15.194:0/2058147343 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f27e0095050 con 0x7f27f0100e00 2024-01-24T14:23:19.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:19 smithi194 ceph-mon[45450]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-01-24T14:23:19.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:19 smithi194 ceph-mon[45450]: osdmap e45: 8 total, 7 up, 8 in 2024-01-24T14:23:19.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:19 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-01-24T14:23:19.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:19 smithi194 ceph-mon[45450]: from='osd.7 [v2:172.21.15.123:6816/1391966446,v1:172.21.15.123:6817/1391966446]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-01-24T14:23:19.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:19 smithi194 ceph-mon[45450]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-01-24T14:23:19.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:19 smithi194 ceph-mon[45450]: Detected new or changed devices on smithi123 2024-01-24T14:23:19.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:19 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:19.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:19 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:19.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:19 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:23:19.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:19 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:23:19.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:19 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:23:19.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:19 smithi194 ceph-mon[45450]: Adjusting osd_memory_target on smithi123 to 5704M 2024-01-24T14:23:19.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:19 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:19.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:19 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:23:19.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:19 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:23:19.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:19 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:19.383 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:19.383+0000 7f27f8498700 1 -- 172.21.15.194:0/2058147343 --> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f27f0061180 con 0x7f27dc0a0520 2024-01-24T14:23:19.485 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:19 smithi019 ceph-mon[40663]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-01-24T14:23:19.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:19 smithi019 ceph-mon[40663]: osdmap e45: 8 total, 7 up, 8 in 2024-01-24T14:23:19.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:19 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-01-24T14:23:19.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:19 smithi019 ceph-mon[40663]: from='osd.7 [v2:172.21.15.123:6816/1391966446,v1:172.21.15.123:6817/1391966446]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-01-24T14:23:19.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:19 smithi019 ceph-mon[40663]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-01-24T14:23:19.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:19 smithi019 ceph-mon[40663]: Detected new or changed devices on smithi123 2024-01-24T14:23:19.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:19 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:19.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:19 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:19.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:19 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:23:19.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:19 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:23:19.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:19 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:23:19.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:19 smithi019 ceph-mon[40663]: Adjusting osd_memory_target on smithi123 to 5704M 2024-01-24T14:23:19.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:19 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:19.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:19 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:23:19.487 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:19 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:23:19.487 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:19 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:19.549 INFO:journalctl@ceph.osd.7.smithi123.stdout:Jan 24 14:23:19 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-7[66344]: 2024-01-24T14:23:19.068+0000 7fd8d4deb700 -1 osd.7 0 waiting for initial osdmap 2024-01-24T14:23:19.550 INFO:journalctl@ceph.osd.7.smithi123.stdout:Jan 24 14:23:19 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-7[66344]: 2024-01-24T14:23:19.079+0000 7fd8cfbde700 -1 osd.7 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-24T14:23:19.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:19 smithi123 ceph-mon[44966]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-01-24T14:23:19.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:19 smithi123 ceph-mon[44966]: osdmap e45: 8 total, 7 up, 8 in 2024-01-24T14:23:19.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:19 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-01-24T14:23:19.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:19 smithi123 ceph-mon[44966]: from='osd.7 [v2:172.21.15.123:6816/1391966446,v1:172.21.15.123:6817/1391966446]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-01-24T14:23:19.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:19 smithi123 ceph-mon[44966]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-01-24T14:23:19.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:19 smithi123 ceph-mon[44966]: Detected new or changed devices on smithi123 2024-01-24T14:23:19.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:19 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:19.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:19 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:19.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:19 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:23:19.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:19 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:23:19.551 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:19 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:23:19.551 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:19 smithi123 ceph-mon[44966]: Adjusting osd_memory_target on smithi123 to 5704M 2024-01-24T14:23:19.551 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:19 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:19.551 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:19 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:23:19.551 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:19 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:23:19.551 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:19 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:20.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:20 smithi194 ceph-mon[45450]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T14:23:20.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:20 smithi194 ceph-mon[45450]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-01-24T14:23:20.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:20 smithi194 ceph-mon[45450]: osdmap e46: 8 total, 7 up, 8 in 2024-01-24T14:23:20.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:20 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-01-24T14:23:20.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:20 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-01-24T14:23:20.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:20 smithi194 ceph-mon[45450]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:23:20.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:20 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T14:23:20.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:20 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T14:23:20.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:20 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:23:20.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:20 smithi194 ceph-mon[45450]: osd.7 [v2:172.21.15.123:6816/1391966446,v1:172.21.15.123:6817/1391966446] boot 2024-01-24T14:23:20.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:20 smithi194 ceph-mon[45450]: osdmap e47: 8 total, 8 up, 8 in 2024-01-24T14:23:20.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:20 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-01-24T14:23:20.485 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:20 smithi019 ceph-mon[40663]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T14:23:20.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:20 smithi019 ceph-mon[40663]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-01-24T14:23:20.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:20 smithi019 ceph-mon[40663]: osdmap e46: 8 total, 7 up, 8 in 2024-01-24T14:23:20.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:20 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-01-24T14:23:20.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:20 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-01-24T14:23:20.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:20 smithi019 ceph-mon[40663]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:23:20.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:20 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T14:23:20.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:20 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T14:23:20.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:20 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:23:20.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:20 smithi019 ceph-mon[40663]: osd.7 [v2:172.21.15.123:6816/1391966446,v1:172.21.15.123:6817/1391966446] boot 2024-01-24T14:23:20.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:20 smithi019 ceph-mon[40663]: osdmap e47: 8 total, 8 up, 8 in 2024-01-24T14:23:20.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:20 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-01-24T14:23:20.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:20 smithi123 ceph-mon[44966]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 188 MiB used, 626 GiB / 626 GiB avail 2024-01-24T14:23:20.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:20 smithi123 ceph-mon[44966]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-01-24T14:23:20.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:20 smithi123 ceph-mon[44966]: osdmap e46: 8 total, 7 up, 8 in 2024-01-24T14:23:20.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:20 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-01-24T14:23:20.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:20 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-01-24T14:23:20.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:20 smithi123 ceph-mon[44966]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:23:20.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:20 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T14:23:20.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:20 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T14:23:20.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:20 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:23:20.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:20 smithi123 ceph-mon[44966]: osd.7 [v2:172.21.15.123:6816/1391966446,v1:172.21.15.123:6817/1391966446] boot 2024-01-24T14:23:20.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:20 smithi123 ceph-mon[44966]: osdmap e47: 8 total, 8 up, 8 in 2024-01-24T14:23:20.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:20 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-01-24T14:23:21.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:21 smithi019 ceph-mon[40663]: purged_snaps scrub starts 2024-01-24T14:23:21.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:21 smithi019 ceph-mon[40663]: purged_snaps scrub ok 2024-01-24T14:23:21.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:21 smithi019 ceph-mon[40663]: osdmap e48: 8 total, 8 up, 8 in 2024-01-24T14:23:21.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:21 smithi019 ceph-mon[40663]: from='client.? 172.21.15.194:0/3468126842' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e918ec1b-7608-4f73-9067-29082ea63999"}]: dispatch 2024-01-24T14:23:21.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:21 smithi019 ceph-mon[40663]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e918ec1b-7608-4f73-9067-29082ea63999"}]: dispatch 2024-01-24T14:23:21.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:21 smithi019 ceph-mon[40663]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e918ec1b-7608-4f73-9067-29082ea63999"}]': finished 2024-01-24T14:23:21.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:21 smithi019 ceph-mon[40663]: osdmap e49: 9 total, 8 up, 9 in 2024-01-24T14:23:21.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:21 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T14:23:21.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:21 smithi123 ceph-mon[44966]: purged_snaps scrub starts 2024-01-24T14:23:21.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:21 smithi123 ceph-mon[44966]: purged_snaps scrub ok 2024-01-24T14:23:21.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:21 smithi123 ceph-mon[44966]: osdmap e48: 8 total, 8 up, 8 in 2024-01-24T14:23:21.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:21 smithi123 ceph-mon[44966]: from='client.? 172.21.15.194:0/3468126842' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e918ec1b-7608-4f73-9067-29082ea63999"}]: dispatch 2024-01-24T14:23:21.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:21 smithi123 ceph-mon[44966]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e918ec1b-7608-4f73-9067-29082ea63999"}]: dispatch 2024-01-24T14:23:21.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:21 smithi123 ceph-mon[44966]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e918ec1b-7608-4f73-9067-29082ea63999"}]': finished 2024-01-24T14:23:21.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:21 smithi123 ceph-mon[44966]: osdmap e49: 9 total, 8 up, 9 in 2024-01-24T14:23:21.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:21 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T14:23:21.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:21 smithi194 ceph-mon[45450]: purged_snaps scrub starts 2024-01-24T14:23:21.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:21 smithi194 ceph-mon[45450]: purged_snaps scrub ok 2024-01-24T14:23:21.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:21 smithi194 ceph-mon[45450]: osdmap e48: 8 total, 8 up, 8 in 2024-01-24T14:23:21.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:21 smithi194 ceph-mon[45450]: from='client.? 172.21.15.194:0/3468126842' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e918ec1b-7608-4f73-9067-29082ea63999"}]: dispatch 2024-01-24T14:23:21.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:21 smithi194 ceph-mon[45450]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e918ec1b-7608-4f73-9067-29082ea63999"}]: dispatch 2024-01-24T14:23:21.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:21 smithi194 ceph-mon[45450]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e918ec1b-7608-4f73-9067-29082ea63999"}]': finished 2024-01-24T14:23:21.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:21 smithi194 ceph-mon[45450]: osdmap e49: 9 total, 8 up, 9 in 2024-01-24T14:23:21.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:21 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T14:23:22.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:22 smithi019 ceph-mon[40663]: pgmap v137: 1 pgs: 1 remapped; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T14:23:22.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:22 smithi019 ceph-mon[40663]: from='client.? 172.21.15.194:0/2426958308' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T14:23:22.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:22 smithi019 ceph-mon[40663]: osdmap e50: 9 total, 8 up, 9 in 2024-01-24T14:23:22.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:22 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T14:23:22.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:22 smithi123 ceph-mon[44966]: pgmap v137: 1 pgs: 1 remapped; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T14:23:22.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:22 smithi123 ceph-mon[44966]: from='client.? 172.21.15.194:0/2426958308' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T14:23:22.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:22 smithi123 ceph-mon[44966]: osdmap e50: 9 total, 8 up, 9 in 2024-01-24T14:23:22.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:22 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T14:23:22.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:22 smithi194 ceph-mon[45450]: pgmap v137: 1 pgs: 1 remapped; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T14:23:22.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:22 smithi194 ceph-mon[45450]: from='client.? 172.21.15.194:0/2426958308' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T14:23:22.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:22 smithi194 ceph-mon[45450]: osdmap e50: 9 total, 8 up, 9 in 2024-01-24T14:23:22.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:22 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T14:23:24.485 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:24 smithi019 ceph-mon[40663]: pgmap v140: 1 pgs: 1 remapped; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T14:23:24.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:24 smithi019 ceph-mon[40663]: osdmap e51: 9 total, 8 up, 9 in 2024-01-24T14:23:24.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:24 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T14:23:24.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:24 smithi123 ceph-mon[44966]: pgmap v140: 1 pgs: 1 remapped; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T14:23:24.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:24 smithi123 ceph-mon[44966]: osdmap e51: 9 total, 8 up, 9 in 2024-01-24T14:23:24.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:24 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T14:23:24.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:24 smithi194 ceph-mon[45450]: pgmap v140: 1 pgs: 1 remapped; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T14:23:24.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:24 smithi194 ceph-mon[45450]: osdmap e51: 9 total, 8 up, 9 in 2024-01-24T14:23:24.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:24 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T14:23:26.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:26 smithi194 ceph-mon[45450]: pgmap v142: 1 pgs: 1 remapped; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T14:23:26.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:26 smithi019 ceph-mon[40663]: pgmap v142: 1 pgs: 1 remapped; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T14:23:26.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:26 smithi123 ceph-mon[44966]: pgmap v142: 1 pgs: 1 remapped; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T14:23:27.718 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:27 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-01-24T14:23:27.718 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:27 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:23:27.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:27 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-01-24T14:23:27.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:27 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:23:27.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:27 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-01-24T14:23:27.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:27 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:23:28.696 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:28 smithi194 ceph-mon[45450]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T14:23:28.696 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:28 smithi194 ceph-mon[45450]: Deploying daemon osd.8 on smithi194 2024-01-24T14:23:28.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:28 smithi019 ceph-mon[40663]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T14:23:28.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:28 smithi019 ceph-mon[40663]: Deploying daemon osd.8 on smithi194 2024-01-24T14:23:28.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:28 smithi123 ceph-mon[44966]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T14:23:28.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:28 smithi123 ceph-mon[44966]: Deploying daemon osd.8 on smithi194 2024-01-24T14:23:30.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:30 smithi019 ceph-mon[40663]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T14:23:30.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:30 smithi123 ceph-mon[44966]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T14:23:30.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:30 smithi194 ceph-mon[45450]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T14:23:31.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:31 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:31.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:31 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:31.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:31 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:23:31.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:31 smithi194 ceph-mon[45450]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T14:23:31.985 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:31 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:31.985 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:31 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:31.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:31 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:23:31.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:31 smithi019 ceph-mon[40663]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T14:23:32.049 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:31 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:32.049 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:31 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:32.049 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:31 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:23:32.049 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:31 smithi123 ceph-mon[44966]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T14:23:32.991 INFO:teuthology.orchestra.run.smithi194.stdout:Created osd(s) 8 on host 'smithi194' 2024-01-24T14:23:32.992 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:32.990+0000 7f27e77fe700 1 -- 172.21.15.194:0/2058147343 <== mgr.14152 v2:172.21.15.19:6800/1634350318 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f27f0061180 con 0x7f27dc0a0520 2024-01-24T14:23:32.993 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:32.992+0000 7f27f8498700 1 -- 172.21.15.194:0/2058147343 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f27dc0a0520 msgr2=0x7f27dc0a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:23:32.993 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:32.992+0000 7f27f8498700 1 --2- 172.21.15.194:0/2058147343 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f27dc0a0520 0x7f27dc0a29e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f27ec007900 tx=0x7f27ec008040 comp rx=0 tx=0).stop 2024-01-24T14:23:32.994 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:32.992+0000 7f27f8498700 1 -- 172.21.15.194:0/2058147343 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f27f0100e00 msgr2=0x7f27f0074ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:23:32.994 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:32.992+0000 7f27f8498700 1 --2- 172.21.15.194:0/2058147343 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f27f0100e00 0x7f27f0074ae0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f27e000b530 tx=0x7f27e000b8f0 comp rx=0 tx=0).stop 2024-01-24T14:23:32.994 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:32.992+0000 7f27f8498700 1 -- 172.21.15.194:0/2058147343 shutdown_connections 2024-01-24T14:23:32.994 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:32.992+0000 7f27f8498700 1 --2- 172.21.15.194:0/2058147343 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f27dc0a0520 0x7f27dc0a29e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:23:32.994 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:32.992+0000 7f27f8498700 1 --2- 172.21.15.194:0/2058147343 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f27f0102920 0x7f27f0073670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:23:32.995 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:32.992+0000 7f27f8498700 1 --2- 172.21.15.194:0/2058147343 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f27f0100e00 0x7f27f0074ae0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:23:32.995 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:32.992+0000 7f27f8498700 1 --2- 172.21.15.194:0/2058147343 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27f0101f60 0x7f27f0073130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:23:32.995 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:32.993+0000 7f27f8498700 1 -- 172.21.15.194:0/2058147343 >> 172.21.15.194:0/2058147343 conn(0x7f27f00fc360 msgr2=0x7f27f0105530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:23:32.995 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:32.993+0000 7f27f8498700 1 -- 172.21.15.194:0/2058147343 shutdown_connections 2024-01-24T14:23:32.995 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:23:32.993+0000 7f27f8498700 1 -- 172.21.15.194:0/2058147343 wait complete. 2024-01-24T14:23:33.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:32 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:33.095 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:32 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:33.095 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:32 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:23:33.095 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:32 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:23:33.095 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:32 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:33.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:32 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:33.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:32 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:33.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:32 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:23:33.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:32 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:23:33.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:32 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:33.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:32 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:33.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:32 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:33.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:32 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:23:33.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:32 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:23:33.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:32 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:33.376 DEBUG:teuthology.orchestra.run.smithi194:osd.8> sudo journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@osd.8.service 2024-01-24T14:23:33.379 INFO:tasks.cephadm:Deploying osd.9 on smithi019 with /dev/vg_nvme/lv_1... 2024-01-24T14:23:33.379 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-01-24T14:23:33.410 INFO:journalctl@ceph.osd.8.smithi194.stdout:-- Logs begin at Wed 2024-01-24 14:04:01 UTC. -- 2024-01-24T14:23:33.649 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/mon.a/config 2024-01-24T14:23:34.063 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:33 smithi019 ceph-mon[40663]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T14:23:34.063 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:33 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:34.063 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:33 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:34.063 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:33 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:23:34.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:33 smithi123 ceph-mon[44966]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T14:23:34.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:33 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:34.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:33 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:34.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:33 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:23:34.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:33 smithi194 ceph-mon[45450]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T14:23:34.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:33 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:34.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:33 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:34.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:33 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:23:34.963 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:23:34 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:23:34.634+0000 7fc9e4783640 -1 osd.8 0 log_to_monitors true 2024-01-24T14:23:34.976 INFO:teuthology.orchestra.run.smithi019.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-01-24T14:23:34.977 INFO:teuthology.orchestra.run.smithi019.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-01-24T14:23:34.977 INFO:teuthology.orchestra.run.smithi019.stderr: stderr: 10+0 records in 2024-01-24T14:23:34.977 INFO:teuthology.orchestra.run.smithi019.stderr:10+0 records out 2024-01-24T14:23:34.977 INFO:teuthology.orchestra.run.smithi019.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0122281 s, 858 MB/s 2024-01-24T14:23:34.977 INFO:teuthology.orchestra.run.smithi019.stderr:--> Zapping successful for: 2024-01-24T14:23:35.334 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 -- ceph orch daemon add osd smithi019:vg_nvme/lv_1 2024-01-24T14:23:35.587 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/mon.a/config 2024-01-24T14:23:35.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:35 smithi194 ceph-mon[45450]: from='osd.8 [v2:172.21.15.194:6816/2260577139,v1:172.21.15.194:6817/2260577139]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-01-24T14:23:35.714 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:35 smithi194 ceph-mon[45450]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-01-24T14:23:35.715 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:35 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:35.715 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:35 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:35.715 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:35 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:23:35.715 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:35 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:23:35.715 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:35 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:23:35.715 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:35 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:35.715 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:35 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:23:35.715 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:35 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:23:35.715 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:35 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:35.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:35 smithi019 ceph-mon[40663]: from='osd.8 [v2:172.21.15.194:6816/2260577139,v1:172.21.15.194:6817/2260577139]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-01-24T14:23:35.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:35 smithi019 ceph-mon[40663]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-01-24T14:23:35.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:35 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:35.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:35 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:35.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:35 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:23:35.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:35 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:23:35.716 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:35 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:23:35.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:35 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:35.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:35 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:23:35.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:35 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:23:35.717 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:35 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:35.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:35 smithi123 ceph-mon[44966]: from='osd.8 [v2:172.21.15.194:6816/2260577139,v1:172.21.15.194:6817/2260577139]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-01-24T14:23:35.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:35 smithi123 ceph-mon[44966]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-01-24T14:23:35.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:35 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:35.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:35 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:35.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:35 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:23:35.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:35 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:23:35.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:35 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:23:35.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:35 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:35.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:35 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:23:35.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:35 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:23:35.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:35 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:36.616 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:36.615+0000 7ff28243b700 1 -- 172.21.15.19:0/2171044913 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff27c101870 msgr2=0x7ff27c108120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:23:36.617 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:36.615+0000 7ff28243b700 1 --2- 172.21.15.19:0/2171044913 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff27c101870 0x7ff27c108120 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff27400b0d0 tx=0x7ff27400b3e0 comp rx=0 tx=0).stop 2024-01-24T14:23:36.617 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:36.616+0000 7ff28243b700 1 -- 172.21.15.19:0/2171044913 shutdown_connections 2024-01-24T14:23:36.617 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:36.616+0000 7ff28243b700 1 --2- 172.21.15.19:0/2171044913 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff27c101870 0x7ff27c108120 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:23:36.617 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:36.616+0000 7ff28243b700 1 --2- 172.21.15.19:0/2171044913 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff27c0ffcd0 0x7ff27c1000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:23:36.617 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:36.616+0000 7ff28243b700 1 --2- 172.21.15.19:0/2171044913 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff27c100ed0 0x7ff27c101330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:23:36.617 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:36.616+0000 7ff28243b700 1 -- 172.21.15.19:0/2171044913 >> 172.21.15.19:0/2171044913 conn(0x7ff27c0fb250 msgr2=0x7ff27c0fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:23:36.617 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:36.616+0000 7ff28243b700 1 -- 172.21.15.19:0/2171044913 shutdown_connections 2024-01-24T14:23:36.617 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:36.616+0000 7ff28243b700 1 -- 172.21.15.19:0/2171044913 wait complete. 2024-01-24T14:23:36.617 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:36.616+0000 7ff28243b700 1 Processor -- start 2024-01-24T14:23:36.617 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:36.616+0000 7ff28243b700 1 -- start start 2024-01-24T14:23:36.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:36.617+0000 7ff28243b700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff27c0ffcd0 0x7ff27c069360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:23:36.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:36.617+0000 7ff28243b700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff27c100ed0 0x7ff27c19dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:23:36.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:36.617+0000 7ff28243b700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff27c101870 0x7ff27c1a0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:23:36.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:36.617+0000 7ff28243b700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff27c1a08e0 con 0x7ff27c100ed0 2024-01-24T14:23:36.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:36.617+0000 7ff28243b700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff27c1a0a50 con 0x7ff27c0ffcd0 2024-01-24T14:23:36.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:36.617+0000 7ff28243b700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff27c1a0c10 con 0x7ff27c101870 2024-01-24T14:23:36.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:36.617+0000 7ff27b7fe700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff27c100ed0 0x7ff27c19dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:23:36.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:36.617+0000 7ff27b7fe700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff27c100ed0 0x7ff27c19dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:47778/0 (socket says 172.21.15.19:47778) 2024-01-24T14:23:36.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:36.617+0000 7ff27b7fe700 1 -- 172.21.15.19:0/1450827807 learned_addr learned my addr 172.21.15.19:0/1450827807 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:23:36.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:36.617+0000 7ff27bfff700 1 --2- 172.21.15.19:0/1450827807 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff27c0ffcd0 0x7ff27c069360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:23:36.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:36.617+0000 7ff2809d8700 1 --2- 172.21.15.19:0/1450827807 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff27c101870 0x7ff27c1a0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:23:36.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:36.617+0000 7ff27b7fe700 1 -- 172.21.15.19:0/1450827807 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff27c101870 msgr2=0x7ff27c1a0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:23:36.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:36.617+0000 7ff27b7fe700 1 --2- 172.21.15.19:0/1450827807 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff27c101870 0x7ff27c1a0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:23:36.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:36.617+0000 7ff27b7fe700 1 -- 172.21.15.19:0/1450827807 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff27c0ffcd0 msgr2=0x7ff27c069360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:23:36.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:36.617+0000 7ff27b7fe700 1 --2- 172.21.15.19:0/1450827807 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff27c0ffcd0 0x7ff27c069360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:23:36.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:36.617+0000 7ff27b7fe700 1 -- 172.21.15.19:0/1450827807 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff274009d00 con 0x7ff27c100ed0 2024-01-24T14:23:36.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:36.618+0000 7ff27b7fe700 1 --2- 172.21.15.19:0/1450827807 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff27c100ed0 0x7ff27c19dce0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff26400d950 tx=0x7ff26400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:23:36.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:36.618+0000 7ff2797fa700 1 -- 172.21.15.19:0/1450827807 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff264009940 con 0x7ff27c100ed0 2024-01-24T14:23:36.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:36.618+0000 7ff2797fa700 1 -- 172.21.15.19:0/1450827807 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff264010460 con 0x7ff27c100ed0 2024-01-24T14:23:36.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:36.618+0000 7ff28243b700 1 -- 172.21.15.19:0/1450827807 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff27c1a0ef0 con 0x7ff27c100ed0 2024-01-24T14:23:36.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:36.618+0000 7ff2797fa700 1 -- 172.21.15.19:0/1450827807 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26400f620 con 0x7ff27c100ed0 2024-01-24T14:23:36.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:36.618+0000 7ff28243b700 1 -- 172.21.15.19:0/1450827807 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff27c1a1360 con 0x7ff27c100ed0 2024-01-24T14:23:36.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:36.619+0000 7ff28243b700 1 -- 172.21.15.19:0/1450827807 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff27c107220 con 0x7ff27c100ed0 2024-01-24T14:23:36.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:36.620+0000 7ff2797fa700 1 -- 172.21.15.19:0/1450827807 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7ff2640108d0 con 0x7ff27c100ed0 2024-01-24T14:23:36.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:36.621+0000 7ff2797fa700 1 --2- 172.21.15.19:0/1450827807 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7ff2680a0520 0x7ff2680a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:23:36.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:36.621+0000 7ff2797fa700 1 -- 172.21.15.19:0/1450827807 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5549+0+0 (secure 0 0 0) 0x7ff2640cacf0 con 0x7ff27c100ed0 2024-01-24T14:23:36.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:36.621+0000 7ff27bfff700 1 --2- 172.21.15.19:0/1450827807 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7ff2680a0520 0x7ff2680a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:23:36.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:36.621+0000 7ff27bfff700 1 --2- 172.21.15.19:0/1450827807 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7ff2680a0520 0x7ff2680a29e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff27c100d30 tx=0x7ff26c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:23:36.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:36.624+0000 7ff2797fa700 1 -- 172.21.15.19:0/1450827807 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7ff264095360 con 0x7ff27c100ed0 2024-01-24T14:23:36.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:36 smithi019 ceph-mon[40663]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T14:23:36.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:36 smithi019 ceph-mon[40663]: Detected new or changed devices on smithi194 2024-01-24T14:23:36.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:36 smithi019 ceph-mon[40663]: Adjusting osd_memory_target on smithi194 to 5704M 2024-01-24T14:23:36.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:36 smithi019 ceph-mon[40663]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-01-24T14:23:36.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:36 smithi019 ceph-mon[40663]: osdmap e52: 9 total, 8 up, 9 in 2024-01-24T14:23:36.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:36 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T14:23:36.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:36 smithi019 ceph-mon[40663]: from='osd.8 [v2:172.21.15.194:6816/2260577139,v1:172.21.15.194:6817/2260577139]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-01-24T14:23:36.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:36 smithi019 ceph-mon[40663]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-01-24T14:23:36.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:36 smithi123 ceph-mon[44966]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T14:23:36.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:36 smithi123 ceph-mon[44966]: Detected new or changed devices on smithi194 2024-01-24T14:23:36.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:36 smithi123 ceph-mon[44966]: Adjusting osd_memory_target on smithi194 to 5704M 2024-01-24T14:23:36.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:36 smithi123 ceph-mon[44966]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-01-24T14:23:36.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:36 smithi123 ceph-mon[44966]: osdmap e52: 9 total, 8 up, 9 in 2024-01-24T14:23:36.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:36 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T14:23:36.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:36 smithi123 ceph-mon[44966]: from='osd.8 [v2:172.21.15.194:6816/2260577139,v1:172.21.15.194:6817/2260577139]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-01-24T14:23:36.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:36 smithi123 ceph-mon[44966]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-01-24T14:23:36.839 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:36.839+0000 7ff28243b700 1 -- 172.21.15.19:0/1450827807 --> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7ff27c061180 con 0x7ff2680a0520 2024-01-24T14:23:36.844 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:23:36 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:23:36.430+0000 7fc9da5e9700 -1 osd.8 0 waiting for initial osdmap 2024-01-24T14:23:36.844 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:23:36 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:23:36.443+0000 7fc9d33d8700 -1 osd.8 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-24T14:23:36.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:36 smithi194 ceph-mon[45450]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T14:23:36.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:36 smithi194 ceph-mon[45450]: Detected new or changed devices on smithi194 2024-01-24T14:23:36.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:36 smithi194 ceph-mon[45450]: Adjusting osd_memory_target on smithi194 to 5704M 2024-01-24T14:23:36.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:36 smithi194 ceph-mon[45450]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-01-24T14:23:36.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:36 smithi194 ceph-mon[45450]: osdmap e52: 9 total, 8 up, 9 in 2024-01-24T14:23:36.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:36 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T14:23:36.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:36 smithi194 ceph-mon[45450]: from='osd.8 [v2:172.21.15.194:6816/2260577139,v1:172.21.15.194:6817/2260577139]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-01-24T14:23:36.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:36 smithi194 ceph-mon[45450]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-01-24T14:23:37.587 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:37 smithi019 ceph-mon[40663]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-01-24T14:23:37.588 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:37 smithi019 ceph-mon[40663]: osdmap e53: 9 total, 8 up, 9 in 2024-01-24T14:23:37.588 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:37 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T14:23:37.588 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:37 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T14:23:37.588 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:37 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T14:23:37.588 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:37 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T14:23:37.588 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:37 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:23:37.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:37 smithi123 ceph-mon[44966]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-01-24T14:23:37.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:37 smithi123 ceph-mon[44966]: osdmap e53: 9 total, 8 up, 9 in 2024-01-24T14:23:37.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:37 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T14:23:37.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:37 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T14:23:37.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:37 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T14:23:37.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:37 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T14:23:37.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:37 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:23:37.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:37 smithi194 ceph-mon[45450]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-01-24T14:23:37.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:37 smithi194 ceph-mon[45450]: osdmap e53: 9 total, 8 up, 9 in 2024-01-24T14:23:37.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:37 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T14:23:37.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:37 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T14:23:37.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:37 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T14:23:37.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:37 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T14:23:37.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:37 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:23:38.741 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:38 smithi019 ceph-mon[40663]: purged_snaps scrub starts 2024-01-24T14:23:38.741 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:38 smithi019 ceph-mon[40663]: purged_snaps scrub ok 2024-01-24T14:23:38.741 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:38 smithi019 ceph-mon[40663]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:23:38.741 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:38 smithi019 ceph-mon[40663]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T14:23:38.741 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:38 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T14:23:38.741 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:38 smithi019 ceph-mon[40663]: osd.8 [v2:172.21.15.194:6816/2260577139,v1:172.21.15.194:6817/2260577139] boot 2024-01-24T14:23:38.741 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:38 smithi019 ceph-mon[40663]: osdmap e54: 9 total, 9 up, 9 in 2024-01-24T14:23:38.741 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:38 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T14:23:38.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:38 smithi123 ceph-mon[44966]: purged_snaps scrub starts 2024-01-24T14:23:38.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:38 smithi123 ceph-mon[44966]: purged_snaps scrub ok 2024-01-24T14:23:38.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:38 smithi123 ceph-mon[44966]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:23:38.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:38 smithi123 ceph-mon[44966]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T14:23:38.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:38 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T14:23:38.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:38 smithi123 ceph-mon[44966]: osd.8 [v2:172.21.15.194:6816/2260577139,v1:172.21.15.194:6817/2260577139] boot 2024-01-24T14:23:38.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:38 smithi123 ceph-mon[44966]: osdmap e54: 9 total, 9 up, 9 in 2024-01-24T14:23:38.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:38 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T14:23:38.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:38 smithi194 ceph-mon[45450]: purged_snaps scrub starts 2024-01-24T14:23:38.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:38 smithi194 ceph-mon[45450]: purged_snaps scrub ok 2024-01-24T14:23:38.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:38 smithi194 ceph-mon[45450]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi019:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:23:38.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:38 smithi194 ceph-mon[45450]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 715 GiB / 715 GiB avail 2024-01-24T14:23:38.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:38 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T14:23:38.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:38 smithi194 ceph-mon[45450]: osd.8 [v2:172.21.15.194:6816/2260577139,v1:172.21.15.194:6817/2260577139] boot 2024-01-24T14:23:38.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:38 smithi194 ceph-mon[45450]: osdmap e54: 9 total, 9 up, 9 in 2024-01-24T14:23:38.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:38 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-01-24T14:23:39.738 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:39 smithi019 ceph-mon[40663]: osdmap e55: 9 total, 9 up, 9 in 2024-01-24T14:23:39.738 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:39 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/423384635' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0fd76722-3319-4828-ab68-108607a4bd0c"}]: dispatch 2024-01-24T14:23:39.738 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:39 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/423384635' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0fd76722-3319-4828-ab68-108607a4bd0c"}]': finished 2024-01-24T14:23:39.738 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:39 smithi019 ceph-mon[40663]: osdmap e56: 10 total, 9 up, 10 in 2024-01-24T14:23:39.738 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:39 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-01-24T14:23:39.739 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:39 smithi019 ceph-mon[40663]: osdmap e57: 10 total, 9 up, 10 in 2024-01-24T14:23:39.739 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:39 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-01-24T14:23:39.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:39 smithi123 ceph-mon[44966]: osdmap e55: 9 total, 9 up, 9 in 2024-01-24T14:23:39.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:39 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/423384635' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0fd76722-3319-4828-ab68-108607a4bd0c"}]: dispatch 2024-01-24T14:23:39.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:39 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/423384635' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0fd76722-3319-4828-ab68-108607a4bd0c"}]': finished 2024-01-24T14:23:39.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:39 smithi123 ceph-mon[44966]: osdmap e56: 10 total, 9 up, 10 in 2024-01-24T14:23:39.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:39 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-01-24T14:23:39.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:39 smithi123 ceph-mon[44966]: osdmap e57: 10 total, 9 up, 10 in 2024-01-24T14:23:39.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:39 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-01-24T14:23:39.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:39 smithi194 ceph-mon[45450]: osdmap e55: 9 total, 9 up, 9 in 2024-01-24T14:23:39.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:39 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/423384635' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0fd76722-3319-4828-ab68-108607a4bd0c"}]: dispatch 2024-01-24T14:23:39.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:39 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/423384635' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0fd76722-3319-4828-ab68-108607a4bd0c"}]': finished 2024-01-24T14:23:39.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:39 smithi194 ceph-mon[45450]: osdmap e56: 10 total, 9 up, 10 in 2024-01-24T14:23:39.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:39 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-01-24T14:23:39.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:39 smithi194 ceph-mon[45450]: osdmap e57: 10 total, 9 up, 10 in 2024-01-24T14:23:39.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:39 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-01-24T14:23:40.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:40 smithi019 ceph-mon[40663]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 804 GiB / 805 GiB avail 2024-01-24T14:23:40.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:40 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/3107998156' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T14:23:40.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:40 smithi019 ceph-mon[40663]: osdmap e58: 10 total, 9 up, 10 in 2024-01-24T14:23:40.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:40 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-01-24T14:23:40.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:40 smithi123 ceph-mon[44966]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 804 GiB / 805 GiB avail 2024-01-24T14:23:40.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:40 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/3107998156' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T14:23:40.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:40 smithi123 ceph-mon[44966]: osdmap e58: 10 total, 9 up, 10 in 2024-01-24T14:23:40.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:40 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-01-24T14:23:40.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:40 smithi194 ceph-mon[45450]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 804 GiB / 805 GiB avail 2024-01-24T14:23:40.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:40 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/3107998156' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T14:23:40.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:40 smithi194 ceph-mon[45450]: osdmap e58: 10 total, 9 up, 10 in 2024-01-24T14:23:40.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:40 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-01-24T14:23:42.712 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:42 smithi019 ceph-mon[40663]: pgmap v157: 1 pgs: 1 peering; 577 KiB data, 242 MiB used, 804 GiB / 805 GiB avail; 164 KiB/s, 0 objects/s recovering 2024-01-24T14:23:42.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:42 smithi123 ceph-mon[44966]: pgmap v157: 1 pgs: 1 peering; 577 KiB data, 242 MiB used, 804 GiB / 805 GiB avail; 164 KiB/s, 0 objects/s recovering 2024-01-24T14:23:42.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:42 smithi194 ceph-mon[45450]: pgmap v157: 1 pgs: 1 peering; 577 KiB data, 242 MiB used, 804 GiB / 805 GiB avail; 164 KiB/s, 0 objects/s recovering 2024-01-24T14:23:44.666 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:44 smithi019 ceph-mon[40663]: pgmap v158: 1 pgs: 1 peering; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 127 KiB/s, 0 objects/s recovering 2024-01-24T14:23:44.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:44 smithi123 ceph-mon[44966]: pgmap v158: 1 pgs: 1 peering; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 127 KiB/s, 0 objects/s recovering 2024-01-24T14:23:44.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:44 smithi194 ceph-mon[45450]: pgmap v158: 1 pgs: 1 peering; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 127 KiB/s, 0 objects/s recovering 2024-01-24T14:23:45.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:45 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-01-24T14:23:45.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:45 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:23:45.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:45 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-01-24T14:23:45.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:45 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:23:45.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:45 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-01-24T14:23:45.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:45 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:23:46.526 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:46 smithi019 ceph-mon[40663]: Deploying daemon osd.9 on smithi019 2024-01-24T14:23:46.526 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:46 smithi019 ceph-mon[40663]: pgmap v159: 1 pgs: 1 peering; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-01-24T14:23:46.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:46 smithi123 ceph-mon[44966]: Deploying daemon osd.9 on smithi019 2024-01-24T14:23:46.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:46 smithi123 ceph-mon[44966]: pgmap v159: 1 pgs: 1 peering; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-01-24T14:23:46.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:46 smithi194 ceph-mon[45450]: Deploying daemon osd.9 on smithi019 2024-01-24T14:23:46.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:46 smithi194 ceph-mon[45450]: pgmap v159: 1 pgs: 1 peering; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-01-24T14:23:48.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:48 smithi019 ceph-mon[40663]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-01-24T14:23:48.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:48 smithi123 ceph-mon[44966]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-01-24T14:23:48.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:48 smithi194 ceph-mon[45450]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-01-24T14:23:49.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:49 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:49.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:49 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:49.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:49 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:23:49.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:49 smithi019 ceph-mon[40663]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-01-24T14:23:49.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:49 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:49.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:49 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:49.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:49 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:23:49.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:49 smithi194 ceph-mon[45450]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-01-24T14:23:50.049 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:49 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:50.049 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:49 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:50.049 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:49 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:23:50.050 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:49 smithi123 ceph-mon[44966]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-01-24T14:23:50.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:50.889+0000 7ff2797fa700 1 -- 172.21.15.19:0/1450827807 <== mgr.14152 v2:172.21.15.19:6800/1634350318 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff27c061180 con 0x7ff2680a0520 2024-01-24T14:23:50.891 INFO:teuthology.orchestra.run.smithi019.stdout:Created osd(s) 9 on host 'smithi019' 2024-01-24T14:23:50.893 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:50.891+0000 7ff28243b700 1 -- 172.21.15.19:0/1450827807 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7ff2680a0520 msgr2=0x7ff2680a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:23:50.893 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:50.891+0000 7ff28243b700 1 --2- 172.21.15.19:0/1450827807 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7ff2680a0520 0x7ff2680a29e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff27c100d30 tx=0x7ff26c009260 comp rx=0 tx=0).stop 2024-01-24T14:23:50.893 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:50.891+0000 7ff28243b700 1 -- 172.21.15.19:0/1450827807 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff27c100ed0 msgr2=0x7ff27c19dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:23:50.893 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:50.891+0000 7ff28243b700 1 --2- 172.21.15.19:0/1450827807 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff27c100ed0 0x7ff27c19dce0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff26400d950 tx=0x7ff26400dc60 comp rx=0 tx=0).stop 2024-01-24T14:23:50.893 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:50.892+0000 7ff28243b700 1 -- 172.21.15.19:0/1450827807 shutdown_connections 2024-01-24T14:23:50.893 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:50.892+0000 7ff28243b700 1 --2- 172.21.15.19:0/1450827807 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff27c101870 0x7ff27c1a0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:23:50.893 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:50.892+0000 7ff28243b700 1 --2- 172.21.15.19:0/1450827807 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff27c0ffcd0 0x7ff27c069360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:23:50.894 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:50.892+0000 7ff28243b700 1 --2- 172.21.15.19:0/1450827807 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7ff2680a0520 0x7ff2680a29e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:23:50.894 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:50.892+0000 7ff28243b700 1 --2- 172.21.15.19:0/1450827807 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff27c100ed0 0x7ff27c19dce0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:23:50.894 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:50.892+0000 7ff28243b700 1 -- 172.21.15.19:0/1450827807 >> 172.21.15.19:0/1450827807 conn(0x7ff27c0fb250 msgr2=0x7ff27c104360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:23:50.894 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:50.892+0000 7ff28243b700 1 -- 172.21.15.19:0/1450827807 shutdown_connections 2024-01-24T14:23:50.894 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:23:50.892+0000 7ff28243b700 1 -- 172.21.15.19:0/1450827807 wait complete. 2024-01-24T14:23:51.351 DEBUG:teuthology.orchestra.run.smithi019:osd.9> sudo journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@osd.9.service 2024-01-24T14:23:51.353 INFO:tasks.cephadm:Deploying osd.10 on smithi123 with /dev/vg_nvme/lv_1... 2024-01-24T14:23:51.353 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-01-24T14:23:51.390 INFO:journalctl@ceph.osd.9.smithi019.stdout:-- Logs begin at Wed 2024-01-24 14:03:52 UTC. -- 2024-01-24T14:23:51.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:51 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:51.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:51 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:51.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:51 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:23:51.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:51 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:23:51.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:51 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:51.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:51 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:51.487 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:51 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:23:51.487 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:51 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:51.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:51 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:51.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:51 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:51.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:51 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:23:51.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:51 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:23:51.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:51 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:51.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:51 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:51.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:51 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:23:51.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:51 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:51.590 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/mon.b/config 2024-01-24T14:23:51.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:51 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:51.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:51 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:51.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:51 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:23:51.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:51 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:23:51.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:51 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:51.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:51 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:51.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:51 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:23:51.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:51 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:52.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:52 smithi123 ceph-mon[44966]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-01-24T14:23:52.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:52 smithi019 ceph-mon[40663]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-01-24T14:23:52.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:52 smithi194 ceph-mon[45450]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-01-24T14:23:52.854 INFO:teuthology.orchestra.run.smithi123.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-01-24T14:23:52.854 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-01-24T14:23:52.854 INFO:teuthology.orchestra.run.smithi123.stderr: stderr: 10+0 records in 2024-01-24T14:23:52.854 INFO:teuthology.orchestra.run.smithi123.stderr:10+0 records out 2024-01-24T14:23:52.854 INFO:teuthology.orchestra.run.smithi123.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0127703 s, 821 MB/s 2024-01-24T14:23:52.854 INFO:teuthology.orchestra.run.smithi123.stderr:--> Zapping successful for: 2024-01-24T14:23:52.986 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:23:52 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:23:52.567+0000 7f4c1167e640 -1 osd.9 0 log_to_monitors true 2024-01-24T14:23:53.249 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 -- ceph orch daemon add osd smithi123:vg_nvme/lv_1 2024-01-24T14:23:53.468 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/mon.b/config 2024-01-24T14:23:53.477 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:53 smithi019 ceph-mon[40663]: from='osd.9 [v2:172.21.15.19:6826/4004843075,v1:172.21.15.19:6827/4004843075]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-01-24T14:23:53.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:53 smithi123 ceph-mon[44966]: from='osd.9 [v2:172.21.15.19:6826/4004843075,v1:172.21.15.19:6827/4004843075]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-01-24T14:23:53.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:53 smithi194 ceph-mon[45450]: from='osd.9 [v2:172.21.15.19:6826/4004843075,v1:172.21.15.19:6827/4004843075]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-01-24T14:23:54.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:54.473+0000 7f0430acc700 1 -- 172.21.15.123:0/2466665093 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f042c1018c0 msgr2=0x7f042c103cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:23:54.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:54.473+0000 7f0430acc700 1 --2- 172.21.15.123:0/2466665093 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f042c1018c0 0x7f042c103cb0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0414009990 tx=0x7f0414009ca0 comp rx=0 tx=0).stop 2024-01-24T14:23:54.476 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:54.473+0000 7f0430acc700 1 -- 172.21.15.123:0/2466665093 shutdown_connections 2024-01-24T14:23:54.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:54.473+0000 7f0430acc700 1 --2- 172.21.15.123:0/2466665093 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f042c106b20 0x7f042c108f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:23:54.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:54.473+0000 7f0430acc700 1 --2- 172.21.15.123:0/2466665093 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f042c1018c0 0x7f042c103cb0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:23:54.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:54.473+0000 7f0430acc700 1 --2- 172.21.15.123:0/2466665093 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f042c1041f0 0x7f042c1065e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:23:54.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:54.473+0000 7f0430acc700 1 -- 172.21.15.123:0/2466665093 >> 172.21.15.123:0/2466665093 conn(0x7f042c0fb250 msgr2=0x7f042c0fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:23:54.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:54.473+0000 7f0430acc700 1 -- 172.21.15.123:0/2466665093 shutdown_connections 2024-01-24T14:23:54.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:54.473+0000 7f0430acc700 1 -- 172.21.15.123:0/2466665093 wait complete. 2024-01-24T14:23:54.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:54.474+0000 7f0430acc700 1 Processor -- start 2024-01-24T14:23:54.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:54.474+0000 7f0430acc700 1 -- start start 2024-01-24T14:23:54.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:54.474+0000 7f0430acc700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f042c1018c0 0x7f042c196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:23:54.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:54.474+0000 7f0430acc700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f042c1041f0 0x7f042c196f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:23:54.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:54.474+0000 7f0430acc700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f042c106b20 0x7f042c19dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:23:54.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:54.474+0000 7f0430acc700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f042c19e5a0 con 0x7f042c106b20 2024-01-24T14:23:54.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:54.474+0000 7f0430acc700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f042c19e6e0 con 0x7f042c1041f0 2024-01-24T14:23:54.477 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:54.474+0000 7f0430acc700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f042c19e870 con 0x7f042c1018c0 2024-01-24T14:23:54.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:54.474+0000 7f0429d9b700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f042c1041f0 0x7f042c196f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:23:54.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:54.474+0000 7f0429d9b700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f042c1041f0 0x7f042c196f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:52836/0 (socket says 172.21.15.123:52836) 2024-01-24T14:23:54.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:54.474+0000 7f0429d9b700 1 -- 172.21.15.123:0/2635247860 learned_addr learned my addr 172.21.15.123:0/2635247860 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-01-24T14:23:54.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:54.474+0000 7f042a59c700 1 --2- 172.21.15.123:0/2635247860 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f042c1018c0 0x7f042c196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:23:54.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:54.474+0000 7f042ad9d700 1 --2- 172.21.15.123:0/2635247860 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f042c106b20 0x7f042c19dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:23:54.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:54.475+0000 7f0429d9b700 1 -- 172.21.15.123:0/2635247860 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f042c1018c0 msgr2=0x7f042c196a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:23:54.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:54.475+0000 7f0429d9b700 1 --2- 172.21.15.123:0/2635247860 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f042c1018c0 0x7f042c196a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:23:54.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:54.475+0000 7f0429d9b700 1 -- 172.21.15.123:0/2635247860 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f042c106b20 msgr2=0x7f042c19dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:23:54.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:54.475+0000 7f0429d9b700 1 --2- 172.21.15.123:0/2635247860 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f042c106b20 0x7f042c19dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:23:54.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:54.475+0000 7f0429d9b700 1 -- 172.21.15.123:0/2635247860 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04140095f0 con 0x7f042c1041f0 2024-01-24T14:23:54.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:54.475+0000 7f0429d9b700 1 --2- 172.21.15.123:0/2635247860 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f042c1041f0 0x7f042c196f50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f041c00c9b0 tx=0x7f041c00ccc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:23:54.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:54.475+0000 7f04237fe700 1 -- 172.21.15.123:0/2635247860 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f041c007b00 con 0x7f042c1041f0 2024-01-24T14:23:54.478 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:54.475+0000 7f04237fe700 1 -- 172.21.15.123:0/2635247860 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f041c00ce50 con 0x7f042c1041f0 2024-01-24T14:23:54.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:54.475+0000 7f0430acc700 1 -- 172.21.15.123:0/2635247860 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f042c19eb50 con 0x7f042c1041f0 2024-01-24T14:23:54.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:54.475+0000 7f04237fe700 1 -- 172.21.15.123:0/2635247860 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f041c018660 con 0x7f042c1041f0 2024-01-24T14:23:54.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:54.475+0000 7f0430acc700 1 -- 172.21.15.123:0/2635247860 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f042c19f050 con 0x7f042c1041f0 2024-01-24T14:23:54.479 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:54.476+0000 7f0430acc700 1 -- 172.21.15.123:0/2635247860 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f042c10c5f0 con 0x7f042c1041f0 2024-01-24T14:23:54.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:54.477+0000 7f04237fe700 1 -- 172.21.15.123:0/2635247860 <== mon.2 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f041c00f450 con 0x7f042c1041f0 2024-01-24T14:23:54.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:54.477+0000 7f04237fe700 1 --2- 172.21.15.123:0/2635247860 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f04180a0520 0x7f04180a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:23:54.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:54.478+0000 7f04237fe700 1 -- 172.21.15.123:0/2635247860 <== mon.2 v2:172.21.15.123:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5981+0+0 (secure 0 0 0) 0x7f041c0caf60 con 0x7f042c1041f0 2024-01-24T14:23:54.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:54.478+0000 7f042a59c700 1 --2- 172.21.15.123:0/2635247860 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f04180a0520 0x7f04180a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:23:54.480 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:54.478+0000 7f042a59c700 1 --2- 172.21.15.123:0/2635247860 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f04180a0520 0x7f04180a29e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f041400b5c0 tx=0x7f0414005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:23:54.482 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:54.482+0000 7f04237fe700 1 -- 172.21.15.123:0/2635247860 <== mon.2 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f041c095420 con 0x7f042c1041f0 2024-01-24T14:23:54.485 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:23:54 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:23:54.189+0000 7f4c074e4700 -1 osd.9 0 waiting for initial osdmap 2024-01-24T14:23:54.485 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:23:54 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:23:54.207+0000 7f4c01ad6700 -1 osd.9 60 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-24T14:23:54.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:54 smithi019 ceph-mon[40663]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-01-24T14:23:54.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:54 smithi019 ceph-mon[40663]: from='osd.9 [v2:172.21.15.19:6826/4004843075,v1:172.21.15.19:6827/4004843075]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-01-24T14:23:54.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:54 smithi019 ceph-mon[40663]: osdmap e59: 10 total, 9 up, 10 in 2024-01-24T14:23:54.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:54 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-01-24T14:23:54.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:54 smithi019 ceph-mon[40663]: from='osd.9 [v2:172.21.15.19:6826/4004843075,v1:172.21.15.19:6827/4004843075]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-01-24T14:23:54.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:54 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:54.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:54 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:54.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:54 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:23:54.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:54 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:23:54.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:54 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:23:54.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:54 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:23:54.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:54 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:54.487 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:54 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:23:54.487 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:54 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:23:54.487 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:54 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:54.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:54 smithi123 ceph-mon[44966]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-01-24T14:23:54.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:54 smithi123 ceph-mon[44966]: from='osd.9 [v2:172.21.15.19:6826/4004843075,v1:172.21.15.19:6827/4004843075]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-01-24T14:23:54.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:54 smithi123 ceph-mon[44966]: osdmap e59: 10 total, 9 up, 10 in 2024-01-24T14:23:54.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:54 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-01-24T14:23:54.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:54 smithi123 ceph-mon[44966]: from='osd.9 [v2:172.21.15.19:6826/4004843075,v1:172.21.15.19:6827/4004843075]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-01-24T14:23:54.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:54 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:54.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:54 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:54.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:54 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:23:54.551 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:54 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:23:54.551 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:54 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:23:54.551 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:54 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:23:54.551 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:54 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:54.551 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:54 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:23:54.551 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:54 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:23:54.551 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:54 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:54.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:54 smithi194 ceph-mon[45450]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-01-24T14:23:54.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:54 smithi194 ceph-mon[45450]: from='osd.9 [v2:172.21.15.19:6826/4004843075,v1:172.21.15.19:6827/4004843075]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-01-24T14:23:54.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:54 smithi194 ceph-mon[45450]: osdmap e59: 10 total, 9 up, 10 in 2024-01-24T14:23:54.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:54 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-01-24T14:23:54.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:54 smithi194 ceph-mon[45450]: from='osd.9 [v2:172.21.15.19:6826/4004843075,v1:172.21.15.19:6827/4004843075]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]: dispatch 2024-01-24T14:23:54.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:54 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:54.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:54 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:54.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:54 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:23:54.595 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:54 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:23:54.595 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:54 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:23:54.595 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:54 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:23:54.595 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:54 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:54.595 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:54 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:23:54.595 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:54 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:23:54.595 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:54 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:23:54.677 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:23:54.676+0000 7f0430acc700 1 -- 172.21.15.123:0/2635247860 --> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f042c103470 con 0x7f04180a0520 2024-01-24T14:23:55.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:55 smithi123 ceph-mon[44966]: Detected new or changed devices on smithi019 2024-01-24T14:23:55.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:55 smithi123 ceph-mon[44966]: Adjusting osd_memory_target on smithi019 to 4279M 2024-01-24T14:23:55.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:55 smithi123 ceph-mon[44966]: from='osd.9 [v2:172.21.15.19:6826/4004843075,v1:172.21.15.19:6827/4004843075]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]': finished 2024-01-24T14:23:55.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:55 smithi123 ceph-mon[44966]: osdmap e60: 10 total, 9 up, 10 in 2024-01-24T14:23:55.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:55 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-01-24T14:23:55.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:55 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-01-24T14:23:55.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:55 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T14:23:55.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:55 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T14:23:55.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:55 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:23:55.488 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:55 smithi019 ceph-mon[40663]: Detected new or changed devices on smithi019 2024-01-24T14:23:55.489 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:55 smithi019 ceph-mon[40663]: Adjusting osd_memory_target on smithi019 to 4279M 2024-01-24T14:23:55.489 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:55 smithi019 ceph-mon[40663]: from='osd.9 [v2:172.21.15.19:6826/4004843075,v1:172.21.15.19:6827/4004843075]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]': finished 2024-01-24T14:23:55.489 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:55 smithi019 ceph-mon[40663]: osdmap e60: 10 total, 9 up, 10 in 2024-01-24T14:23:55.489 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:55 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-01-24T14:23:55.489 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:55 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-01-24T14:23:55.489 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:55 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T14:23:55.489 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:55 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T14:23:55.489 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:55 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:23:55.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:55 smithi194 ceph-mon[45450]: Detected new or changed devices on smithi019 2024-01-24T14:23:55.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:55 smithi194 ceph-mon[45450]: Adjusting osd_memory_target on smithi019 to 4279M 2024-01-24T14:23:55.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:55 smithi194 ceph-mon[45450]: from='osd.9 [v2:172.21.15.19:6826/4004843075,v1:172.21.15.19:6827/4004843075]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi019", "root=default"]}]': finished 2024-01-24T14:23:55.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:55 smithi194 ceph-mon[45450]: osdmap e60: 10 total, 9 up, 10 in 2024-01-24T14:23:55.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:55 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-01-24T14:23:55.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:55 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-01-24T14:23:55.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:55 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T14:23:55.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:55 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T14:23:55.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:55 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:23:56.485 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:56 smithi019 ceph-mon[40663]: purged_snaps scrub starts 2024-01-24T14:23:56.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:56 smithi019 ceph-mon[40663]: purged_snaps scrub ok 2024-01-24T14:23:56.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:56 smithi019 ceph-mon[40663]: from='client.24356 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:23:56.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:56 smithi019 ceph-mon[40663]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-01-24T14:23:56.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:56 smithi019 ceph-mon[40663]: osd.9 [v2:172.21.15.19:6826/4004843075,v1:172.21.15.19:6827/4004843075] boot 2024-01-24T14:23:56.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:56 smithi019 ceph-mon[40663]: osdmap e61: 10 total, 10 up, 10 in 2024-01-24T14:23:56.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:56 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-01-24T14:23:56.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:56 smithi123 ceph-mon[44966]: purged_snaps scrub starts 2024-01-24T14:23:56.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:56 smithi123 ceph-mon[44966]: purged_snaps scrub ok 2024-01-24T14:23:56.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:56 smithi123 ceph-mon[44966]: from='client.24356 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:23:56.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:56 smithi123 ceph-mon[44966]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-01-24T14:23:56.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:56 smithi123 ceph-mon[44966]: osd.9 [v2:172.21.15.19:6826/4004843075,v1:172.21.15.19:6827/4004843075] boot 2024-01-24T14:23:56.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:56 smithi123 ceph-mon[44966]: osdmap e61: 10 total, 10 up, 10 in 2024-01-24T14:23:56.551 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:56 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-01-24T14:23:56.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:56 smithi194 ceph-mon[45450]: purged_snaps scrub starts 2024-01-24T14:23:56.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:56 smithi194 ceph-mon[45450]: purged_snaps scrub ok 2024-01-24T14:23:56.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:56 smithi194 ceph-mon[45450]: from='client.24356 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:23:56.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:56 smithi194 ceph-mon[45450]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-01-24T14:23:56.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:56 smithi194 ceph-mon[45450]: osd.9 [v2:172.21.15.19:6826/4004843075,v1:172.21.15.19:6827/4004843075] boot 2024-01-24T14:23:56.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:56 smithi194 ceph-mon[45450]: osdmap e61: 10 total, 10 up, 10 in 2024-01-24T14:23:56.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:56 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-01-24T14:23:57.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:57 smithi019 ceph-mon[40663]: osdmap e62: 10 total, 10 up, 10 in 2024-01-24T14:23:57.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:57 smithi019 ceph-mon[40663]: from='client.? 172.21.15.123:0/2934380223' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "624bb8f5-6d72-43d5-9986-1b48b8a564ac"}]: dispatch 2024-01-24T14:23:57.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:57 smithi019 ceph-mon[40663]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "624bb8f5-6d72-43d5-9986-1b48b8a564ac"}]: dispatch 2024-01-24T14:23:57.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:57 smithi019 ceph-mon[40663]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "624bb8f5-6d72-43d5-9986-1b48b8a564ac"}]': finished 2024-01-24T14:23:57.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:57 smithi019 ceph-mon[40663]: osdmap e63: 11 total, 10 up, 11 in 2024-01-24T14:23:57.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:57 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-01-24T14:23:57.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:57 smithi019 ceph-mon[40663]: from='client.? 172.21.15.123:0/513052044' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T14:23:57.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:57 smithi123 ceph-mon[44966]: osdmap e62: 10 total, 10 up, 10 in 2024-01-24T14:23:57.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:57 smithi123 ceph-mon[44966]: from='client.? 172.21.15.123:0/2934380223' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "624bb8f5-6d72-43d5-9986-1b48b8a564ac"}]: dispatch 2024-01-24T14:23:57.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:57 smithi123 ceph-mon[44966]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "624bb8f5-6d72-43d5-9986-1b48b8a564ac"}]: dispatch 2024-01-24T14:23:57.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:57 smithi123 ceph-mon[44966]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "624bb8f5-6d72-43d5-9986-1b48b8a564ac"}]': finished 2024-01-24T14:23:57.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:57 smithi123 ceph-mon[44966]: osdmap e63: 11 total, 10 up, 11 in 2024-01-24T14:23:57.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:57 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-01-24T14:23:57.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:57 smithi123 ceph-mon[44966]: from='client.? 172.21.15.123:0/513052044' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T14:23:57.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:57 smithi194 ceph-mon[45450]: osdmap e62: 10 total, 10 up, 10 in 2024-01-24T14:23:57.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:57 smithi194 ceph-mon[45450]: from='client.? 172.21.15.123:0/2934380223' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "624bb8f5-6d72-43d5-9986-1b48b8a564ac"}]: dispatch 2024-01-24T14:23:57.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:57 smithi194 ceph-mon[45450]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "624bb8f5-6d72-43d5-9986-1b48b8a564ac"}]: dispatch 2024-01-24T14:23:57.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:57 smithi194 ceph-mon[45450]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "624bb8f5-6d72-43d5-9986-1b48b8a564ac"}]': finished 2024-01-24T14:23:57.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:57 smithi194 ceph-mon[45450]: osdmap e63: 11 total, 10 up, 11 in 2024-01-24T14:23:57.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:57 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-01-24T14:23:57.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:57 smithi194 ceph-mon[45450]: from='client.? 172.21.15.123:0/513052044' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T14:23:58.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:23:58 smithi019 ceph-mon[40663]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-01-24T14:23:58.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:23:58 smithi123 ceph-mon[44966]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-01-24T14:23:58.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:23:58 smithi194 ceph-mon[45450]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-01-24T14:24:00.531 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:00 smithi123 ceph-mon[44966]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-01-24T14:24:00.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:00 smithi019 ceph-mon[40663]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-01-24T14:24:00.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:00 smithi194 ceph-mon[45450]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-01-24T14:24:02.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:02 smithi019 ceph-mon[40663]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-01-24T14:24:02.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:02 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-01-24T14:24:02.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:02 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:02.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:02 smithi123 ceph-mon[44966]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-01-24T14:24:02.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:02 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-01-24T14:24:02.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:02 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:02.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:02 smithi194 ceph-mon[45450]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-01-24T14:24:02.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:02 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-01-24T14:24:02.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:02 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:03.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:03 smithi019 ceph-mon[40663]: Deploying daemon osd.10 on smithi123 2024-01-24T14:24:03.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:03 smithi123 ceph-mon[44966]: Deploying daemon osd.10 on smithi123 2024-01-24T14:24:03.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:03 smithi194 ceph-mon[45450]: Deploying daemon osd.10 on smithi123 2024-01-24T14:24:04.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:04 smithi019 ceph-mon[40663]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-01-24T14:24:04.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:04 smithi123 ceph-mon[44966]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-01-24T14:24:04.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:04 smithi194 ceph-mon[45450]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-01-24T14:24:06.725 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:06 smithi123 ceph-mon[44966]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-01-24T14:24:06.725 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:06 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:06.725 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:06 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:06.725 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:06 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:24:06.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:06 smithi019 ceph-mon[40663]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-01-24T14:24:06.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:06 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:06.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:06 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:06.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:06 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:24:06.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:06 smithi194 ceph-mon[45450]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-01-24T14:24:06.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:06 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:06.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:06 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:06.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:06 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:24:08.383 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:24:08.380+0000 7f04237fe700 1 -- 172.21.15.123:0/2635247860 <== mgr.14152 v2:172.21.15.19:6800/1634350318 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f042c103470 con 0x7f04180a0520 2024-01-24T14:24:08.384 INFO:teuthology.orchestra.run.smithi123.stdout:Created osd(s) 10 on host 'smithi123' 2024-01-24T14:24:08.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:24:08.382+0000 7f0430acc700 1 -- 172.21.15.123:0/2635247860 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f04180a0520 msgr2=0x7f04180a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:08.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:24:08.383+0000 7f0430acc700 1 --2- 172.21.15.123:0/2635247860 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f04180a0520 0x7f04180a29e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f041400b5c0 tx=0x7f0414005960 comp rx=0 tx=0).stop 2024-01-24T14:24:08.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:24:08.383+0000 7f0430acc700 1 -- 172.21.15.123:0/2635247860 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f042c1041f0 msgr2=0x7f042c196f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:08.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:24:08.383+0000 7f0430acc700 1 --2- 172.21.15.123:0/2635247860 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f042c1041f0 0x7f042c196f50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f041c00c9b0 tx=0x7f041c00ccc0 comp rx=0 tx=0).stop 2024-01-24T14:24:08.386 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:24:08.383+0000 7f0430acc700 1 -- 172.21.15.123:0/2635247860 shutdown_connections 2024-01-24T14:24:08.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:24:08.383+0000 7f0430acc700 1 --2- 172.21.15.123:0/2635247860 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f04180a0520 0x7f04180a29e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:08.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:24:08.383+0000 7f0430acc700 1 --2- 172.21.15.123:0/2635247860 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f042c106b20 0x7f042c19dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:08.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:24:08.383+0000 7f0430acc700 1 --2- 172.21.15.123:0/2635247860 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f042c1018c0 0x7f042c196a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:08.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:24:08.383+0000 7f0430acc700 1 --2- 172.21.15.123:0/2635247860 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f042c1041f0 0x7f042c196f50 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:08.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:24:08.383+0000 7f0430acc700 1 -- 172.21.15.123:0/2635247860 >> 172.21.15.123:0/2635247860 conn(0x7f042c0fb250 msgr2=0x7f042c102d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:24:08.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:24:08.383+0000 7f0430acc700 1 -- 172.21.15.123:0/2635247860 shutdown_connections 2024-01-24T14:24:08.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-01-24T14:24:08.383+0000 7f0430acc700 1 -- 172.21.15.123:0/2635247860 wait complete. 2024-01-24T14:24:08.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:08 smithi123 ceph-mon[44966]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-01-24T14:24:08.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:08 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:08.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:08 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:08.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:08 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:08.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:08 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:24:08.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:08 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:08.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:08 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:08.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:08 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:24:08.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:08 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:08.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:08 smithi019 ceph-mon[40663]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-01-24T14:24:08.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:08 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:08.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:08 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:08.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:08 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:08.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:08 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:24:08.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:08 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:08.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:08 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:08.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:08 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:24:08.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:08 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:08.776 DEBUG:teuthology.orchestra.run.smithi123:osd.10> sudo journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@osd.10.service 2024-01-24T14:24:08.778 INFO:tasks.cephadm:Deploying osd.11 on smithi194 with /dev/vg_nvme/lv_1... 2024-01-24T14:24:08.778 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-01-24T14:24:08.808 INFO:journalctl@ceph.osd.10.smithi123.stdout:-- Logs begin at Wed 2024-01-24 14:03:45 UTC. -- 2024-01-24T14:24:08.811 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:08 smithi194 ceph-mon[45450]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-01-24T14:24:08.811 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:08 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:08.811 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:08 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:08.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:08 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:08.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:08 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:24:08.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:08 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:08.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:08 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:08.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:08 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:24:08.812 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:08 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:09.010 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/mon.c/config 2024-01-24T14:24:10.260 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-01-24T14:24:10.260 INFO:teuthology.orchestra.run.smithi194.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-01-24T14:24:10.260 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10+0 records in 2024-01-24T14:24:10.260 INFO:teuthology.orchestra.run.smithi194.stderr:10+0 records out 2024-01-24T14:24:10.260 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0144343 s, 726 MB/s 2024-01-24T14:24:10.260 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping successful for: 2024-01-24T14:24:10.300 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:24:10 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:24:10.026+0000 7f6bab1b0640 -1 osd.10 0 log_to_monitors true 2024-01-24T14:24:10.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:10 smithi194 ceph-mon[45450]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-01-24T14:24:10.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:10 smithi194 ceph-mon[45450]: from='osd.10 [v2:172.21.15.123:6824/2591845068,v1:172.21.15.123:6825/2591845068]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-01-24T14:24:10.572 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:10 smithi194 ceph-mon[45450]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-01-24T14:24:10.657 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 -- ceph orch daemon add osd smithi194:vg_nvme/lv_1 2024-01-24T14:24:10.726 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:10 smithi123 ceph-mon[44966]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-01-24T14:24:10.726 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:10 smithi123 ceph-mon[44966]: from='osd.10 [v2:172.21.15.123:6824/2591845068,v1:172.21.15.123:6825/2591845068]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-01-24T14:24:10.726 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:10 smithi123 ceph-mon[44966]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-01-24T14:24:10.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:10 smithi019 ceph-mon[40663]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-01-24T14:24:10.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:10 smithi019 ceph-mon[40663]: from='osd.10 [v2:172.21.15.123:6824/2591845068,v1:172.21.15.123:6825/2591845068]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-01-24T14:24:10.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:10 smithi019 ceph-mon[40663]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-01-24T14:24:10.924 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/mon.c/config 2024-01-24T14:24:11.595 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:11 smithi194 ceph-mon[45450]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-01-24T14:24:11.595 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:11 smithi194 ceph-mon[45450]: osdmap e64: 11 total, 10 up, 11 in 2024-01-24T14:24:11.595 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:11 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-01-24T14:24:11.595 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:11 smithi194 ceph-mon[45450]: from='osd.10 [v2:172.21.15.123:6824/2591845068,v1:172.21.15.123:6825/2591845068]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-01-24T14:24:11.596 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:11 smithi194 ceph-mon[45450]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-01-24T14:24:11.596 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:11 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:11.596 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:11 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:11.596 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:11 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:24:11.596 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:11 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:24:11.596 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:11 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:24:11.597 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:11 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:24:11.597 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:11 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:11.597 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:11 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:11.597 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:11 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:24:11.597 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:11 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:11.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:11 smithi019 ceph-mon[40663]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-01-24T14:24:11.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:11 smithi019 ceph-mon[40663]: osdmap e64: 11 total, 10 up, 11 in 2024-01-24T14:24:11.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:11 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-01-24T14:24:11.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:11 smithi019 ceph-mon[40663]: from='osd.10 [v2:172.21.15.123:6824/2591845068,v1:172.21.15.123:6825/2591845068]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-01-24T14:24:11.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:11 smithi019 ceph-mon[40663]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-01-24T14:24:11.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:11 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:11.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:11 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:11.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:11 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:24:11.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:11 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:24:11.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:11 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:24:11.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:11 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:24:11.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:11 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:11.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:11 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:11.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:11 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:24:11.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:11 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:11.801 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:24:11 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:24:11.461+0000 7f6ba0815700 -1 osd.10 0 waiting for initial osdmap 2024-01-24T14:24:11.801 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:24:11 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:24:11.479+0000 7f6b9a606700 -1 osd.10 65 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-24T14:24:11.802 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:11 smithi123 ceph-mon[44966]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-01-24T14:24:11.802 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:11 smithi123 ceph-mon[44966]: osdmap e64: 11 total, 10 up, 11 in 2024-01-24T14:24:11.802 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:11 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-01-24T14:24:11.802 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:11 smithi123 ceph-mon[44966]: from='osd.10 [v2:172.21.15.123:6824/2591845068,v1:172.21.15.123:6825/2591845068]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-01-24T14:24:11.802 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:11 smithi123 ceph-mon[44966]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-01-24T14:24:11.802 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:11 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:11.802 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:11 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:11.802 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:11 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:24:11.802 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:11 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:24:11.803 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:11 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:24:11.803 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:11 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:24:11.803 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:11 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:11.803 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:11 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:11.803 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:11 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:24:11.803 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:11 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:11.875 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:11.872+0000 7fc6d6746700 1 -- 172.21.15.194:0/3910841156 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc6d00ffd00 msgr2=0x7fc6d0100120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:11.875 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:11.872+0000 7fc6d6746700 1 --2- 172.21.15.194:0/3910841156 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc6d00ffd00 0x7fc6d0100120 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc6c0009990 tx=0x7fc6c0009ca0 comp rx=0 tx=0).stop 2024-01-24T14:24:11.875 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:11.872+0000 7fc6d6746700 1 -- 172.21.15.194:0/3910841156 shutdown_connections 2024-01-24T14:24:11.875 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:11.872+0000 7fc6d6746700 1 --2- 172.21.15.194:0/3910841156 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc6d01018a0 0x7fc6d0108150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:11.875 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:11.872+0000 7fc6d6746700 1 --2- 172.21.15.194:0/3910841156 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc6d00ffd00 0x7fc6d0100120 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:11.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:11.872+0000 7fc6d6746700 1 --2- 172.21.15.194:0/3910841156 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6d0100f00 0x7fc6d0101360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:11.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:11.872+0000 7fc6d6746700 1 -- 172.21.15.194:0/3910841156 >> 172.21.15.194:0/3910841156 conn(0x7fc6d00fb2c0 msgr2=0x7fc6d00fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:24:11.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:11.872+0000 7fc6d6746700 1 -- 172.21.15.194:0/3910841156 shutdown_connections 2024-01-24T14:24:11.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:11.872+0000 7fc6d6746700 1 -- 172.21.15.194:0/3910841156 wait complete. 2024-01-24T14:24:11.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:11.873+0000 7fc6d6746700 1 Processor -- start 2024-01-24T14:24:11.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:11.873+0000 7fc6d6746700 1 -- start start 2024-01-24T14:24:11.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:11.873+0000 7fc6d6746700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc6d00ffd00 0x7fc6d0195ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:11.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:11.873+0000 7fc6d6746700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc6d0100f00 0x7fc6d0196020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:11.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:11.873+0000 7fc6d6746700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6d01018a0 0x7fc6d019d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:11.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:11.873+0000 7fc6d6746700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d019d670 con 0x7fc6d0100f00 2024-01-24T14:24:11.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:11.873+0000 7fc6d6746700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d019d7b0 con 0x7fc6d01018a0 2024-01-24T14:24:11.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:11.873+0000 7fc6d6746700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d019d940 con 0x7fc6d00ffd00 2024-01-24T14:24:11.877 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:11.873+0000 7fc6cffff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc6d00ffd00 0x7fc6d0195ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:11.877 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:11.873+0000 7fc6cffff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc6d00ffd00 0x7fc6d0195ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:43102/0 (socket says 172.21.15.194:43102) 2024-01-24T14:24:11.877 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:11.873+0000 7fc6cffff700 1 -- 172.21.15.194:0/3528940670 learned_addr learned my addr 172.21.15.194:0/3528940670 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-24T14:24:11.877 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:11.874+0000 7fc6cf7fe700 1 --2- 172.21.15.194:0/3528940670 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc6d0100f00 0x7fc6d0196020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:11.877 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:11.874+0000 7fc6cffff700 1 -- 172.21.15.194:0/3528940670 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6d01018a0 msgr2=0x7fc6d019d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:11.877 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:11.874+0000 7fc6cffff700 1 --2- 172.21.15.194:0/3528940670 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6d01018a0 0x7fc6d019d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:11.877 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:11.874+0000 7fc6cffff700 1 -- 172.21.15.194:0/3528940670 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc6d0100f00 msgr2=0x7fc6d0196020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:11.877 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:11.874+0000 7fc6cffff700 1 --2- 172.21.15.194:0/3528940670 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc6d0100f00 0x7fc6d0196020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:11.877 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:11.874+0000 7fc6cffff700 1 -- 172.21.15.194:0/3528940670 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6c00095f0 con 0x7fc6d00ffd00 2024-01-24T14:24:11.877 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:11.874+0000 7fc6cf7fe700 1 --2- 172.21.15.194:0/3528940670 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc6d0100f00 0x7fc6d0196020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T14:24:11.877 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:11.874+0000 7fc6cffff700 1 --2- 172.21.15.194:0/3528940670 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc6d00ffd00 0x7fc6d0195ae0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc6c00098e0 tx=0x7fc6c000f7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:24:11.879 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:11.874+0000 7fc6cd7fa700 1 -- 172.21.15.194:0/3528940670 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c001d070 con 0x7fc6d00ffd00 2024-01-24T14:24:11.879 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:11.874+0000 7fc6cd7fa700 1 -- 172.21.15.194:0/3528940670 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc6c0003e20 con 0x7fc6d00ffd00 2024-01-24T14:24:11.879 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:11.874+0000 7fc6d6746700 1 -- 172.21.15.194:0/3528940670 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6d019dbc0 con 0x7fc6d00ffd00 2024-01-24T14:24:11.880 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:11.874+0000 7fc6cd7fa700 1 -- 172.21.15.194:0/3528940670 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c0017650 con 0x7fc6d00ffd00 2024-01-24T14:24:11.880 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:11.874+0000 7fc6d6746700 1 -- 172.21.15.194:0/3528940670 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6d019e050 con 0x7fc6d00ffd00 2024-01-24T14:24:11.880 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:11.875+0000 7fc6d6746700 1 -- 172.21.15.194:0/3528940670 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6d010b5d0 con 0x7fc6d00ffd00 2024-01-24T14:24:11.880 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:11.876+0000 7fc6cd7fa700 1 -- 172.21.15.194:0/3528940670 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7fc6c0003960 con 0x7fc6d00ffd00 2024-01-24T14:24:11.880 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:11.877+0000 7fc6cd7fa700 1 --2- 172.21.15.194:0/3528940670 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fc6b80a0520 0x7fc6b80a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:11.880 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:11.877+0000 7fc6cd7fa700 1 -- 172.21.15.194:0/3528940670 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fc6c00cc6b0 con 0x7fc6d00ffd00 2024-01-24T14:24:11.880 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:11.877+0000 7fc6cf7fe700 1 --2- 172.21.15.194:0/3528940670 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fc6b80a0520 0x7fc6b80a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:11.880 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:11.877+0000 7fc6cf7fe700 1 --2- 172.21.15.194:0/3528940670 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fc6b80a0520 0x7fc6b80a29e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc6c4007900 tx=0x7fc6c4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:24:11.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:11.881+0000 7fc6cd7fa700 1 -- 172.21.15.194:0/3528940670 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fc6c0096a50 con 0x7fc6d00ffd00 2024-01-24T14:24:12.075 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:12.075+0000 7fc6d6746700 1 -- 172.21.15.194:0/3528940670 --> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fc6d0061180 con 0x7fc6b80a0520 2024-01-24T14:24:12.735 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:12 smithi194 ceph-mon[45450]: Detected new or changed devices on smithi123 2024-01-24T14:24:12.735 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:12 smithi194 ceph-mon[45450]: Adjusting osd_memory_target on smithi123 to 4278M 2024-01-24T14:24:12.735 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:12 smithi194 ceph-mon[45450]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-01-24T14:24:12.735 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:12 smithi194 ceph-mon[45450]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-01-24T14:24:12.735 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:12 smithi194 ceph-mon[45450]: osdmap e65: 11 total, 10 up, 11 in 2024-01-24T14:24:12.735 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:12 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-01-24T14:24:12.735 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:12 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-01-24T14:24:12.736 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:12 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T14:24:12.736 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:12 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T14:24:12.736 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:12 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:12.736 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:12 smithi194 ceph-mon[45450]: osd.10 [v2:172.21.15.123:6824/2591845068,v1:172.21.15.123:6825/2591845068] boot 2024-01-24T14:24:12.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:12 smithi019 ceph-mon[40663]: Detected new or changed devices on smithi123 2024-01-24T14:24:12.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:12 smithi019 ceph-mon[40663]: Adjusting osd_memory_target on smithi123 to 4278M 2024-01-24T14:24:12.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:12 smithi019 ceph-mon[40663]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-01-24T14:24:12.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:12 smithi019 ceph-mon[40663]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-01-24T14:24:12.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:12 smithi019 ceph-mon[40663]: osdmap e65: 11 total, 10 up, 11 in 2024-01-24T14:24:12.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:12 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-01-24T14:24:12.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:12 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-01-24T14:24:12.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:12 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T14:24:12.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:12 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T14:24:12.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:12 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:12.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:12 smithi019 ceph-mon[40663]: osd.10 [v2:172.21.15.123:6824/2591845068,v1:172.21.15.123:6825/2591845068] boot 2024-01-24T14:24:12.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:12 smithi123 ceph-mon[44966]: Detected new or changed devices on smithi123 2024-01-24T14:24:12.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:12 smithi123 ceph-mon[44966]: Adjusting osd_memory_target on smithi123 to 4278M 2024-01-24T14:24:12.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:12 smithi123 ceph-mon[44966]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 894 GiB / 894 GiB avail 2024-01-24T14:24:12.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:12 smithi123 ceph-mon[44966]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-01-24T14:24:12.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:12 smithi123 ceph-mon[44966]: osdmap e65: 11 total, 10 up, 11 in 2024-01-24T14:24:12.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:12 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-01-24T14:24:12.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:12 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-01-24T14:24:12.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:12 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-01-24T14:24:12.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:12 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-01-24T14:24:12.801 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:12 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:12.801 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:12 smithi123 ceph-mon[44966]: osd.10 [v2:172.21.15.123:6824/2591845068,v1:172.21.15.123:6825/2591845068] boot 2024-01-24T14:24:13.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:13 smithi019 ceph-mon[40663]: purged_snaps scrub starts 2024-01-24T14:24:13.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:13 smithi019 ceph-mon[40663]: purged_snaps scrub ok 2024-01-24T14:24:13.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:13 smithi019 ceph-mon[40663]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:24:13.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:13 smithi019 ceph-mon[40663]: osdmap e66: 11 total, 11 up, 11 in 2024-01-24T14:24:13.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:13 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-01-24T14:24:13.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:13 smithi123 ceph-mon[44966]: purged_snaps scrub starts 2024-01-24T14:24:13.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:13 smithi123 ceph-mon[44966]: purged_snaps scrub ok 2024-01-24T14:24:13.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:13 smithi123 ceph-mon[44966]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:24:13.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:13 smithi123 ceph-mon[44966]: osdmap e66: 11 total, 11 up, 11 in 2024-01-24T14:24:13.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:13 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-01-24T14:24:13.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:13 smithi194 ceph-mon[45450]: purged_snaps scrub starts 2024-01-24T14:24:13.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:13 smithi194 ceph-mon[45450]: purged_snaps scrub ok 2024-01-24T14:24:13.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:13 smithi194 ceph-mon[45450]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:24:13.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:13 smithi194 ceph-mon[45450]: osdmap e66: 11 total, 11 up, 11 in 2024-01-24T14:24:13.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:13 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-01-24T14:24:14.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:14 smithi019 ceph-mon[40663]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 983 GiB / 983 GiB avail 2024-01-24T14:24:14.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:14 smithi019 ceph-mon[40663]: osdmap e67: 11 total, 11 up, 11 in 2024-01-24T14:24:14.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:14 smithi019 ceph-mon[40663]: from='client.? 172.21.15.194:0/4289902850' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "219a55b3-b01e-471d-90da-1a3043bf3e34"}]: dispatch 2024-01-24T14:24:14.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:14 smithi019 ceph-mon[40663]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "219a55b3-b01e-471d-90da-1a3043bf3e34"}]: dispatch 2024-01-24T14:24:14.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:14 smithi019 ceph-mon[40663]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "219a55b3-b01e-471d-90da-1a3043bf3e34"}]': finished 2024-01-24T14:24:14.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:14 smithi019 ceph-mon[40663]: osdmap e68: 12 total, 11 up, 12 in 2024-01-24T14:24:14.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:14 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-01-24T14:24:14.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:14 smithi019 ceph-mon[40663]: from='client.? 172.21.15.194:0/883589470' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T14:24:14.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:14 smithi019 ceph-mon[40663]: osdmap e69: 12 total, 11 up, 12 in 2024-01-24T14:24:14.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:14 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-01-24T14:24:14.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:14 smithi123 ceph-mon[44966]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 983 GiB / 983 GiB avail 2024-01-24T14:24:14.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:14 smithi123 ceph-mon[44966]: osdmap e67: 11 total, 11 up, 11 in 2024-01-24T14:24:14.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:14 smithi123 ceph-mon[44966]: from='client.? 172.21.15.194:0/4289902850' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "219a55b3-b01e-471d-90da-1a3043bf3e34"}]: dispatch 2024-01-24T14:24:14.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:14 smithi123 ceph-mon[44966]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "219a55b3-b01e-471d-90da-1a3043bf3e34"}]: dispatch 2024-01-24T14:24:14.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:14 smithi123 ceph-mon[44966]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "219a55b3-b01e-471d-90da-1a3043bf3e34"}]': finished 2024-01-24T14:24:14.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:14 smithi123 ceph-mon[44966]: osdmap e68: 12 total, 11 up, 12 in 2024-01-24T14:24:14.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:14 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-01-24T14:24:14.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:14 smithi123 ceph-mon[44966]: from='client.? 172.21.15.194:0/883589470' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T14:24:14.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:14 smithi123 ceph-mon[44966]: osdmap e69: 12 total, 11 up, 12 in 2024-01-24T14:24:14.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:14 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-01-24T14:24:14.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:14 smithi194 ceph-mon[45450]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 983 GiB / 983 GiB avail 2024-01-24T14:24:14.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:14 smithi194 ceph-mon[45450]: osdmap e67: 11 total, 11 up, 11 in 2024-01-24T14:24:14.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:14 smithi194 ceph-mon[45450]: from='client.? 172.21.15.194:0/4289902850' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "219a55b3-b01e-471d-90da-1a3043bf3e34"}]: dispatch 2024-01-24T14:24:14.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:14 smithi194 ceph-mon[45450]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "219a55b3-b01e-471d-90da-1a3043bf3e34"}]: dispatch 2024-01-24T14:24:14.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:14 smithi194 ceph-mon[45450]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "219a55b3-b01e-471d-90da-1a3043bf3e34"}]': finished 2024-01-24T14:24:14.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:14 smithi194 ceph-mon[45450]: osdmap e68: 12 total, 11 up, 12 in 2024-01-24T14:24:14.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:14 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-01-24T14:24:14.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:14 smithi194 ceph-mon[45450]: from='client.? 172.21.15.194:0/883589470' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-01-24T14:24:14.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:14 smithi194 ceph-mon[45450]: osdmap e69: 12 total, 11 up, 12 in 2024-01-24T14:24:14.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:14 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-01-24T14:24:16.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:16 smithi019 ceph-mon[40663]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 983 GiB / 983 GiB avail 2024-01-24T14:24:16.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:16 smithi019 ceph-mon[40663]: osdmap e70: 12 total, 11 up, 12 in 2024-01-24T14:24:16.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:16 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-01-24T14:24:16.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:16 smithi123 ceph-mon[44966]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 983 GiB / 983 GiB avail 2024-01-24T14:24:16.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:16 smithi123 ceph-mon[44966]: osdmap e70: 12 total, 11 up, 12 in 2024-01-24T14:24:16.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:16 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-01-24T14:24:16.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:16 smithi194 ceph-mon[45450]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 983 GiB / 983 GiB avail 2024-01-24T14:24:16.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:16 smithi194 ceph-mon[45450]: osdmap e70: 12 total, 11 up, 12 in 2024-01-24T14:24:16.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:16 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-01-24T14:24:18.501 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:18 smithi194 ceph-mon[45450]: pgmap v187: 1 pgs: 1 peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-01-24T14:24:18.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:18 smithi019 ceph-mon[40663]: pgmap v187: 1 pgs: 1 peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-01-24T14:24:18.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:18 smithi123 ceph-mon[44966]: pgmap v187: 1 pgs: 1 peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-01-24T14:24:20.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:20 smithi194 ceph-mon[45450]: pgmap v188: 1 pgs: 1 peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-01-24T14:24:20.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:20 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-01-24T14:24:20.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:20 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:20.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:20 smithi019 ceph-mon[40663]: pgmap v188: 1 pgs: 1 peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-01-24T14:24:20.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:20 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-01-24T14:24:20.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:20 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:20.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:20 smithi123 ceph-mon[44966]: pgmap v188: 1 pgs: 1 peering; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-01-24T14:24:20.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:20 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-01-24T14:24:20.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:20 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:21.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:21 smithi194 ceph-mon[45450]: Deploying daemon osd.11 on smithi194 2024-01-24T14:24:21.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:21 smithi019 ceph-mon[40663]: Deploying daemon osd.11 on smithi194 2024-01-24T14:24:21.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:21 smithi123 ceph-mon[44966]: Deploying daemon osd.11 on smithi194 2024-01-24T14:24:22.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:22 smithi019 ceph-mon[40663]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-01-24T14:24:22.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:22 smithi123 ceph-mon[44966]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-01-24T14:24:22.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:22 smithi194 ceph-mon[45450]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-01-24T14:24:24.485 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:24 smithi019 ceph-mon[40663]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-01-24T14:24:24.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:24 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:24.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:24 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:24.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:24 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:24:24.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:24 smithi123 ceph-mon[44966]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-01-24T14:24:24.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:24 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:24.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:24 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:24.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:24 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:24:24.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:24 smithi194 ceph-mon[45450]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-01-24T14:24:24.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:24 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:24.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:24 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:24.594 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:24 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:24:25.530 INFO:teuthology.orchestra.run.smithi194.stdout:Created osd(s) 11 on host 'smithi194' 2024-01-24T14:24:25.531 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:25.529+0000 7fc6cd7fa700 1 -- 172.21.15.194:0/3528940670 <== mgr.14152 v2:172.21.15.19:6800/1634350318 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fc6d0061180 con 0x7fc6b80a0520 2024-01-24T14:24:25.532 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:25.532+0000 7fc6d6746700 1 -- 172.21.15.194:0/3528940670 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fc6b80a0520 msgr2=0x7fc6b80a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:25.532 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:25.532+0000 7fc6d6746700 1 --2- 172.21.15.194:0/3528940670 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fc6b80a0520 0x7fc6b80a29e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc6c4007900 tx=0x7fc6c4008040 comp rx=0 tx=0).stop 2024-01-24T14:24:25.532 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:25.532+0000 7fc6d6746700 1 -- 172.21.15.194:0/3528940670 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc6d00ffd00 msgr2=0x7fc6d0195ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:25.532 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:25.532+0000 7fc6d6746700 1 --2- 172.21.15.194:0/3528940670 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc6d00ffd00 0x7fc6d0195ae0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc6c00098e0 tx=0x7fc6c000f7b0 comp rx=0 tx=0).stop 2024-01-24T14:24:25.533 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:25.532+0000 7fc6d6746700 1 -- 172.21.15.194:0/3528940670 shutdown_connections 2024-01-24T14:24:25.533 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:25.532+0000 7fc6d6746700 1 --2- 172.21.15.194:0/3528940670 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc6d01018a0 0x7fc6d019d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:25.533 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:25.532+0000 7fc6d6746700 1 --2- 172.21.15.194:0/3528940670 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc6d00ffd00 0x7fc6d0195ae0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:25.533 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:25.532+0000 7fc6d6746700 1 --2- 172.21.15.194:0/3528940670 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fc6b80a0520 0x7fc6b80a29e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:25.533 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:25.532+0000 7fc6d6746700 1 --2- 172.21.15.194:0/3528940670 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc6d0100f00 0x7fc6d0196020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:25.533 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:25.532+0000 7fc6d6746700 1 -- 172.21.15.194:0/3528940670 >> 172.21.15.194:0/3528940670 conn(0x7fc6d00fb2c0 msgr2=0x7fc6d00fd5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:24:25.533 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:25.532+0000 7fc6d6746700 1 -- 172.21.15.194:0/3528940670 shutdown_connections 2024-01-24T14:24:25.533 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:25.532+0000 7fc6d6746700 1 -- 172.21.15.194:0/3528940670 wait complete. 2024-01-24T14:24:25.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:25 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:25.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:25 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:25.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:25 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:25.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:25 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:24:25.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:25 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:25.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:25 smithi194 ceph-mon[45450]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-01-24T14:24:25.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:25 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:25.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:25 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:25.826 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:25 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:24:25.985 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:25 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:25.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:25 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:25.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:25 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:25.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:25 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:24:25.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:25 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:25.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:25 smithi019 ceph-mon[40663]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-01-24T14:24:25.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:25 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:25.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:25 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:25.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:25 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:24:26.004 DEBUG:teuthology.orchestra.run.smithi194:osd.11> sudo journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@osd.11.service 2024-01-24T14:24:26.007 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-01-24T14:24:26.007 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 -- ceph osd stat -f json 2024-01-24T14:24:26.041 INFO:journalctl@ceph.osd.11.smithi194.stdout:-- Logs begin at Wed 2024-01-24 14:04:01 UTC. -- 2024-01-24T14:24:26.049 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:25 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:26.049 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:25 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:26.049 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:25 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:26.049 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:25 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:24:26.049 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:25 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:26.049 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:25 smithi123 ceph-mon[44966]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-01-24T14:24:26.049 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:25 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:26.049 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:25 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:26.050 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:25 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:24:26.267 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/mon.a/config 2024-01-24T14:24:27.242 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.241+0000 7ff380b71700 1 -- 172.21.15.19:0/3871609083 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff37c101f60 msgr2=0x7ff37c1023e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:27.243 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.241+0000 7ff380b71700 1 --2- 172.21.15.19:0/3871609083 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff37c101f60 0x7ff37c1023e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff364009990 tx=0x7ff364009ca0 comp rx=0 tx=0).stop 2024-01-24T14:24:27.243 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.241+0000 7ff380b71700 1 -- 172.21.15.19:0/3871609083 shutdown_connections 2024-01-24T14:24:27.243 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.241+0000 7ff380b71700 1 --2- 172.21.15.19:0/3871609083 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff37c102920 0x7ff37c109240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:27.243 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.241+0000 7ff380b71700 1 --2- 172.21.15.19:0/3871609083 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff37c100e00 0x7ff37c101220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:27.243 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.241+0000 7ff380b71700 1 --2- 172.21.15.19:0/3871609083 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff37c101f60 0x7ff37c1023e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:27.243 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.241+0000 7ff380b71700 1 -- 172.21.15.19:0/3871609083 >> 172.21.15.19:0/3871609083 conn(0x7ff37c0fc360 msgr2=0x7ff37c0fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:24:27.243 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.241+0000 7ff380b71700 1 -- 172.21.15.19:0/3871609083 shutdown_connections 2024-01-24T14:24:27.243 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.241+0000 7ff380b71700 1 -- 172.21.15.19:0/3871609083 wait complete. 2024-01-24T14:24:27.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.242+0000 7ff380b71700 1 Processor -- start 2024-01-24T14:24:27.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.242+0000 7ff380b71700 1 -- start start 2024-01-24T14:24:27.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.242+0000 7ff380b71700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff37c100e00 0x7ff37c196a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:27.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.242+0000 7ff380b71700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff37c101f60 0x7ff37c196fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:27.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.242+0000 7ff380b71700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff37c102920 0x7ff37c19e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:27.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.242+0000 7ff380b71700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff37c19e600 con 0x7ff37c100e00 2024-01-24T14:24:27.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.242+0000 7ff380b71700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff37c19e740 con 0x7ff37c101f60 2024-01-24T14:24:27.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.242+0000 7ff380b71700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff37c19e8d0 con 0x7ff37c102920 2024-01-24T14:24:27.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.243+0000 7ff37a59c700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff37c100e00 0x7ff37c196a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:27.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.243+0000 7ff37ad9d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff37c102920 0x7ff37c19e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:27.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.243+0000 7ff37a59c700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff37c100e00 0x7ff37c196a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:54994/0 (socket says 172.21.15.19:54994) 2024-01-24T14:24:27.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.243+0000 7ff37a59c700 1 -- 172.21.15.19:0/1183286798 learned_addr learned my addr 172.21.15.19:0/1183286798 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:24:27.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.243+0000 7ff371d9b700 1 --2- 172.21.15.19:0/1183286798 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff37c101f60 0x7ff37c196fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:27.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.243+0000 7ff37a59c700 1 -- 172.21.15.19:0/1183286798 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff37c102920 msgr2=0x7ff37c19e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:27.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.243+0000 7ff37a59c700 1 --2- 172.21.15.19:0/1183286798 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff37c102920 0x7ff37c19e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:27.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.243+0000 7ff37a59c700 1 -- 172.21.15.19:0/1183286798 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff37c101f60 msgr2=0x7ff37c196fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:27.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.243+0000 7ff37a59c700 1 --2- 172.21.15.19:0/1183286798 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff37c101f60 0x7ff37c196fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:27.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.243+0000 7ff37a59c700 1 -- 172.21.15.19:0/1183286798 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3640095f0 con 0x7ff37c100e00 2024-01-24T14:24:27.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.243+0000 7ff37ad9d700 1 --2- 172.21.15.19:0/1183286798 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff37c102920 0x7ff37c19e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T14:24:27.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.243+0000 7ff37a59c700 1 --2- 172.21.15.19:0/1183286798 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff37c100e00 0x7ff37c196a70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff36c00e9d0 tx=0x7ff36c00ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:24:27.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.243+0000 7ff373fff700 1 -- 172.21.15.19:0/1183286798 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff36c009790 con 0x7ff37c100e00 2024-01-24T14:24:27.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.243+0000 7ff373fff700 1 -- 172.21.15.19:0/1183286798 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff36c00ee70 con 0x7ff37c100e00 2024-01-24T14:24:27.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.243+0000 7ff380b71700 1 -- 172.21.15.19:0/1183286798 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff37c19ebb0 con 0x7ff37c100e00 2024-01-24T14:24:27.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.243+0000 7ff373fff700 1 -- 172.21.15.19:0/1183286798 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff36c010ea0 con 0x7ff37c100e00 2024-01-24T14:24:27.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.244+0000 7ff380b71700 1 -- 172.21.15.19:0/1183286798 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff37c19f0b0 con 0x7ff37c100e00 2024-01-24T14:24:27.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.245+0000 7ff380b71700 1 -- 172.21.15.19:0/1183286798 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff37c1083c0 con 0x7ff37c100e00 2024-01-24T14:24:27.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.246+0000 7ff373fff700 1 -- 172.21.15.19:0/1183286798 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7ff36c00cbd0 con 0x7ff37c100e00 2024-01-24T14:24:27.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.247+0000 7ff373fff700 1 --2- 172.21.15.19:0/1183286798 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7ff3680a0520 0x7ff3680a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:27.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.247+0000 7ff371d9b700 1 --2- 172.21.15.19:0/1183286798 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7ff3680a0520 0x7ff3680a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:27.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.247+0000 7ff373fff700 1 -- 172.21.15.19:0/1183286798 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6809+0+0 (secure 0 0 0) 0x7ff36c014070 con 0x7ff37c100e00 2024-01-24T14:24:27.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.247+0000 7ff371d9b700 1 --2- 172.21.15.19:0/1183286798 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7ff3680a0520 0x7ff3680a29e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff36400b2e0 tx=0x7ff364005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:24:27.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.251+0000 7ff373fff700 1 -- 172.21.15.19:0/1183286798 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7ff36c052120 con 0x7ff37c100e00 2024-01-24T14:24:27.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.459+0000 7ff380b71700 1 -- 172.21.15.19:0/1183286798 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7ff37c02cf20 con 0x7ff37c100e00 2024-01-24T14:24:27.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.459+0000 7ff373fff700 1 -- 172.21.15.19:0/1183286798 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v70) v1 ==== 74+0+133 (secure 0 0 0) 0x7ff36c0956d0 con 0x7ff37c100e00 2024-01-24T14:24:27.460 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:24:27.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.462+0000 7ff380b71700 1 -- 172.21.15.19:0/1183286798 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7ff3680a0520 msgr2=0x7ff3680a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:27.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.462+0000 7ff380b71700 1 --2- 172.21.15.19:0/1183286798 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7ff3680a0520 0x7ff3680a29e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff36400b2e0 tx=0x7ff364005960 comp rx=0 tx=0).stop 2024-01-24T14:24:27.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.462+0000 7ff380b71700 1 -- 172.21.15.19:0/1183286798 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff37c100e00 msgr2=0x7ff37c196a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:27.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.462+0000 7ff380b71700 1 --2- 172.21.15.19:0/1183286798 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff37c100e00 0x7ff37c196a70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff36c00e9d0 tx=0x7ff36c00ece0 comp rx=0 tx=0).stop 2024-01-24T14:24:27.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.462+0000 7ff380b71700 1 -- 172.21.15.19:0/1183286798 shutdown_connections 2024-01-24T14:24:27.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.462+0000 7ff380b71700 1 --2- 172.21.15.19:0/1183286798 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7ff3680a0520 0x7ff3680a29e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:27.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.462+0000 7ff380b71700 1 --2- 172.21.15.19:0/1183286798 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff37c100e00 0x7ff37c196a70 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:27.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.462+0000 7ff380b71700 1 --2- 172.21.15.19:0/1183286798 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff37c102920 0x7ff37c19e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:27.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.462+0000 7ff380b71700 1 --2- 172.21.15.19:0/1183286798 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff37c101f60 0x7ff37c196fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:27.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.462+0000 7ff380b71700 1 -- 172.21.15.19:0/1183286798 >> 172.21.15.19:0/1183286798 conn(0x7ff37c0fc360 msgr2=0x7ff37c105500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:24:27.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.462+0000 7ff380b71700 1 -- 172.21.15.19:0/1183286798 shutdown_connections 2024-01-24T14:24:27.464 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:27.462+0000 7ff380b71700 1 -- 172.21.15.19:0/1183286798 wait complete. 2024-01-24T14:24:27.594 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:24:27 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:24:27.205+0000 7f23937b6640 -1 osd.11 0 log_to_monitors true 2024-01-24T14:24:27.860 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":70,"num_osds":12,"num_up_osds":11,"osd_up_since":1706106252,"num_in_osds":12,"osd_in_since":1706106253,"num_remapped_pgs":0} 2024-01-24T14:24:28.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:28 smithi123 ceph-mon[44966]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-01-24T14:24:28.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:28 smithi123 ceph-mon[44966]: from='osd.11 [v2:172.21.15.194:6824/3985496260,v1:172.21.15.194:6825/3985496260]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-01-24T14:24:28.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:28 smithi123 ceph-mon[44966]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-01-24T14:24:28.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:28 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/1183286798' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-24T14:24:28.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:28 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:28.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:28 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:28.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:28 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:24:28.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:28 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:24:28.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:28 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:24:28.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:28 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:24:28.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:28 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:28.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:28 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:28.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:28 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:24:28.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:28 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:28.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:28 smithi194 ceph-mon[45450]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-01-24T14:24:28.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:28 smithi194 ceph-mon[45450]: from='osd.11 [v2:172.21.15.194:6824/3985496260,v1:172.21.15.194:6825/3985496260]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-01-24T14:24:28.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:28 smithi194 ceph-mon[45450]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-01-24T14:24:28.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:28 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/1183286798' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-24T14:24:28.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:28 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:28.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:28 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:28.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:28 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:24:28.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:28 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:24:28.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:28 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:24:28.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:28 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:24:28.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:28 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:28.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:28 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:28.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:28 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:24:28.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:28 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:28.485 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:28 smithi019 ceph-mon[40663]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-01-24T14:24:28.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:28 smithi019 ceph-mon[40663]: from='osd.11 [v2:172.21.15.194:6824/3985496260,v1:172.21.15.194:6825/3985496260]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-01-24T14:24:28.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:28 smithi019 ceph-mon[40663]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-01-24T14:24:28.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:28 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/1183286798' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-24T14:24:28.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:28 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:28.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:28 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:28.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:28 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:24:28.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:28 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:24:28.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:28 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:24:28.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:28 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-01-24T14:24:28.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:28 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:28.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:28 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:28.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:28 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:24:28.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:28 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:28.862 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 -- ceph osd stat -f json 2024-01-24T14:24:29.119 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/mon.a/config 2024-01-24T14:24:29.230 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:29 smithi019 ceph-mon[40663]: Detected new or changed devices on smithi194 2024-01-24T14:24:29.231 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:29 smithi019 ceph-mon[40663]: Adjusting osd_memory_target on smithi194 to 4278M 2024-01-24T14:24:29.231 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:29 smithi019 ceph-mon[40663]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-01-24T14:24:29.231 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:29 smithi019 ceph-mon[40663]: osdmap e71: 12 total, 11 up, 12 in 2024-01-24T14:24:29.231 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:29 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-01-24T14:24:29.231 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:29 smithi019 ceph-mon[40663]: from='osd.11 [v2:172.21.15.194:6824/3985496260,v1:172.21.15.194:6825/3985496260]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-01-24T14:24:29.231 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:29 smithi019 ceph-mon[40663]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-01-24T14:24:29.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:29 smithi123 ceph-mon[44966]: Detected new or changed devices on smithi194 2024-01-24T14:24:29.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:29 smithi123 ceph-mon[44966]: Adjusting osd_memory_target on smithi194 to 4278M 2024-01-24T14:24:29.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:29 smithi123 ceph-mon[44966]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-01-24T14:24:29.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:29 smithi123 ceph-mon[44966]: osdmap e71: 12 total, 11 up, 12 in 2024-01-24T14:24:29.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:29 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-01-24T14:24:29.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:29 smithi123 ceph-mon[44966]: from='osd.11 [v2:172.21.15.194:6824/3985496260,v1:172.21.15.194:6825/3985496260]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-01-24T14:24:29.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:29 smithi123 ceph-mon[44966]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-01-24T14:24:29.344 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:24:29 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:24:29.035+0000 7f2388e1b700 -1 osd.11 0 waiting for initial osdmap 2024-01-24T14:24:29.344 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:24:29 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:24:29.052+0000 7f238240b700 -1 osd.11 72 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-24T14:24:29.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:29 smithi194 ceph-mon[45450]: Detected new or changed devices on smithi194 2024-01-24T14:24:29.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:29 smithi194 ceph-mon[45450]: Adjusting osd_memory_target on smithi194 to 4278M 2024-01-24T14:24:29.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:29 smithi194 ceph-mon[45450]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-01-24T14:24:29.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:29 smithi194 ceph-mon[45450]: osdmap e71: 12 total, 11 up, 12 in 2024-01-24T14:24:29.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:29 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-01-24T14:24:29.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:29 smithi194 ceph-mon[45450]: from='osd.11 [v2:172.21.15.194:6824/3985496260,v1:172.21.15.194:6825/3985496260]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-01-24T14:24:29.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:29 smithi194 ceph-mon[45450]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-01-24T14:24:30.116 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.115+0000 7f6ddbf45700 1 -- 172.21.15.19:0/2951998928 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6dd40ffcd0 msgr2=0x7f6dd41000f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:30.116 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.115+0000 7f6ddbf45700 1 --2- 172.21.15.19:0/2951998928 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6dd40ffcd0 0x7f6dd41000f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6dd00099c0 tx=0x7f6dd0009cd0 comp rx=0 tx=0).stop 2024-01-24T14:24:30.116 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.115+0000 7f6ddbf45700 1 -- 172.21.15.19:0/2951998928 shutdown_connections 2024-01-24T14:24:30.116 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.115+0000 7f6ddbf45700 1 --2- 172.21.15.19:0/2951998928 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6dd40ffcd0 0x7f6dd41000f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:30.117 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.115+0000 7f6ddbf45700 1 --2- 172.21.15.19:0/2951998928 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6dd4101870 0x7f6dd4108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:30.117 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.115+0000 7f6ddbf45700 1 --2- 172.21.15.19:0/2951998928 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6dd4100ed0 0x7f6dd4101330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:30.117 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.115+0000 7f6ddbf45700 1 -- 172.21.15.19:0/2951998928 >> 172.21.15.19:0/2951998928 conn(0x7f6dd40fb250 msgr2=0x7f6dd40fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:24:30.117 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.115+0000 7f6ddbf45700 1 -- 172.21.15.19:0/2951998928 shutdown_connections 2024-01-24T14:24:30.117 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.115+0000 7f6ddbf45700 1 -- 172.21.15.19:0/2951998928 wait complete. 2024-01-24T14:24:30.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.116+0000 7f6ddbf45700 1 Processor -- start 2024-01-24T14:24:30.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.116+0000 7f6ddbf45700 1 -- start start 2024-01-24T14:24:30.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.116+0000 7f6ddbf45700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6dd40ffcd0 0x7f6dd4196a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:30.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.116+0000 7f6ddbf45700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6dd4100ed0 0x7f6dd4196fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:30.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.116+0000 7f6ddbf45700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6dd4101870 0x7f6dd419e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:30.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.116+0000 7f6ddbf45700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dd419e600 con 0x7f6dd4101870 2024-01-24T14:24:30.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.116+0000 7f6ddbf45700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dd419e740 con 0x7f6dd40ffcd0 2024-01-24T14:24:30.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.116+0000 7f6ddbf45700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dd419e8d0 con 0x7f6dd4100ed0 2024-01-24T14:24:30.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.116+0000 7f6dda4e2700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6dd4101870 0x7f6dd419e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:30.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.116+0000 7f6dda4e2700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6dd4101870 0x7f6dd419e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:55008/0 (socket says 172.21.15.19:55008) 2024-01-24T14:24:30.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.116+0000 7f6dda4e2700 1 -- 172.21.15.19:0/248986388 learned_addr learned my addr 172.21.15.19:0/248986388 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:24:30.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.116+0000 7f6dd9ce1700 1 --2- 172.21.15.19:0/248986388 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6dd40ffcd0 0x7f6dd4196a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:30.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.116+0000 7f6dd94e0700 1 --2- 172.21.15.19:0/248986388 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6dd4100ed0 0x7f6dd4196fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:30.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.116+0000 7f6dda4e2700 1 -- 172.21.15.19:0/248986388 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6dd4100ed0 msgr2=0x7f6dd4196fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:30.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.116+0000 7f6dda4e2700 1 --2- 172.21.15.19:0/248986388 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6dd4100ed0 0x7f6dd4196fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:30.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.116+0000 7f6dda4e2700 1 -- 172.21.15.19:0/248986388 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6dd40ffcd0 msgr2=0x7f6dd4196a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:30.120 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.116+0000 7f6dda4e2700 1 --2- 172.21.15.19:0/248986388 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6dd40ffcd0 0x7f6dd4196a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:30.120 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.116+0000 7f6dda4e2700 1 -- 172.21.15.19:0/248986388 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dd00095f0 con 0x7f6dd4101870 2024-01-24T14:24:30.120 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.116+0000 7f6dd9ce1700 1 --2- 172.21.15.19:0/248986388 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6dd40ffcd0 0x7f6dd4196a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T14:24:30.120 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.117+0000 7f6dda4e2700 1 --2- 172.21.15.19:0/248986388 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6dd4101870 0x7f6dd419e030 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc007f10 tx=0x7f6dcc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:24:30.120 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.117+0000 7f6dcaffd700 1 -- 172.21.15.19:0/248986388 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dcc017070 con 0x7f6dd4101870 2024-01-24T14:24:30.120 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.117+0000 7f6dcaffd700 1 -- 172.21.15.19:0/248986388 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6dcc00f040 con 0x7f6dd4101870 2024-01-24T14:24:30.120 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.117+0000 7f6ddbf45700 1 -- 172.21.15.19:0/248986388 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dd419ebb0 con 0x7f6dd4101870 2024-01-24T14:24:30.120 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.117+0000 7f6dcaffd700 1 -- 172.21.15.19:0/248986388 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dcc005740 con 0x7f6dd4101870 2024-01-24T14:24:30.120 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.117+0000 7f6ddbf45700 1 -- 172.21.15.19:0/248986388 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dd419f0b0 con 0x7f6dd4101870 2024-01-24T14:24:30.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.118+0000 7f6ddbf45700 1 -- 172.21.15.19:0/248986388 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dd410b4d0 con 0x7f6dd4101870 2024-01-24T14:24:30.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.119+0000 7f6dcaffd700 1 -- 172.21.15.19:0/248986388 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f6dcc0058a0 con 0x7f6dd4101870 2024-01-24T14:24:30.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.120+0000 7f6dcaffd700 1 --2- 172.21.15.19:0/248986388 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f6dc00a05a0 0x7f6dc00a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:30.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.120+0000 7f6dd9ce1700 1 --2- 172.21.15.19:0/248986388 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f6dc00a05a0 0x7f6dc00a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:30.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.120+0000 7f6dcaffd700 1 -- 172.21.15.19:0/248986388 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f6dcc0cad00 con 0x7f6dd4101870 2024-01-24T14:24:30.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.120+0000 7f6dd9ce1700 1 --2- 172.21.15.19:0/248986388 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f6dc00a05a0 0x7f6dc00a2a60 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6dd000b580 tx=0x7f6dd0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:24:30.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.124+0000 7f6dcaffd700 1 -- 172.21.15.19:0/248986388 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f6dcc094d40 con 0x7f6dd4101870 2024-01-24T14:24:30.235 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:30 smithi019 ceph-mon[40663]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-01-24T14:24:30.235 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:30 smithi019 ceph-mon[40663]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-01-24T14:24:30.235 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:30 smithi019 ceph-mon[40663]: osdmap e72: 12 total, 11 up, 12 in 2024-01-24T14:24:30.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:30 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-01-24T14:24:30.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:30 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-01-24T14:24:30.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:30 smithi123 ceph-mon[44966]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-01-24T14:24:30.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:30 smithi123 ceph-mon[44966]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-01-24T14:24:30.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:30 smithi123 ceph-mon[44966]: osdmap e72: 12 total, 11 up, 12 in 2024-01-24T14:24:30.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:30 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-01-24T14:24:30.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:30 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-01-24T14:24:30.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.336+0000 7f6ddbf45700 1 -- 172.21.15.19:0/248986388 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f6dd410b6c0 con 0x7f6dd4101870 2024-01-24T14:24:30.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.336+0000 7f6dcaffd700 1 -- 172.21.15.19:0/248986388 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v73) v1 ==== 74+0+133 (secure 0 0 0) 0x7f6dcc098360 con 0x7f6dd4101870 2024-01-24T14:24:30.337 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:24:30.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.338+0000 7f6ddbf45700 1 -- 172.21.15.19:0/248986388 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f6dc00a05a0 msgr2=0x7f6dc00a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:30.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.338+0000 7f6ddbf45700 1 --2- 172.21.15.19:0/248986388 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f6dc00a05a0 0x7f6dc00a2a60 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6dd000b580 tx=0x7f6dd0005fb0 comp rx=0 tx=0).stop 2024-01-24T14:24:30.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.338+0000 7f6ddbf45700 1 -- 172.21.15.19:0/248986388 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6dd4101870 msgr2=0x7f6dd419e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:30.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.338+0000 7f6ddbf45700 1 --2- 172.21.15.19:0/248986388 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6dd4101870 0x7f6dd419e030 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc007f10 tx=0x7f6dcc00d3b0 comp rx=0 tx=0).stop 2024-01-24T14:24:30.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.339+0000 7f6ddbf45700 1 -- 172.21.15.19:0/248986388 shutdown_connections 2024-01-24T14:24:30.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.339+0000 7f6ddbf45700 1 --2- 172.21.15.19:0/248986388 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f6dc00a05a0 0x7f6dc00a2a60 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:30.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.339+0000 7f6ddbf45700 1 --2- 172.21.15.19:0/248986388 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6dd4101870 0x7f6dd419e030 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:30.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.339+0000 7f6ddbf45700 1 --2- 172.21.15.19:0/248986388 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6dd40ffcd0 0x7f6dd4196a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:30.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.339+0000 7f6ddbf45700 1 --2- 172.21.15.19:0/248986388 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6dd4100ed0 0x7f6dd4196fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:30.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.339+0000 7f6ddbf45700 1 -- 172.21.15.19:0/248986388 >> 172.21.15.19:0/248986388 conn(0x7f6dd40fb250 msgr2=0x7f6dd4068860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:24:30.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.339+0000 7f6ddbf45700 1 -- 172.21.15.19:0/248986388 shutdown_connections 2024-01-24T14:24:30.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:30.339+0000 7f6ddbf45700 1 -- 172.21.15.19:0/248986388 wait complete. 2024-01-24T14:24:30.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:30 smithi194 ceph-mon[45450]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-01-24T14:24:30.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:30 smithi194 ceph-mon[45450]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-01-24T14:24:30.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:30 smithi194 ceph-mon[45450]: osdmap e72: 12 total, 11 up, 12 in 2024-01-24T14:24:30.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:30 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-01-24T14:24:30.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:30 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-01-24T14:24:30.718 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":73,"num_osds":12,"num_up_osds":12,"osd_up_since":1706106270,"num_in_osds":12,"osd_in_since":1706106253,"num_remapped_pgs":0} 2024-01-24T14:24:30.719 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 -- ceph osd dump --format=json 2024-01-24T14:24:30.971 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/mon.a/config 2024-01-24T14:24:31.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:31 smithi123 ceph-mon[44966]: purged_snaps scrub starts 2024-01-24T14:24:31.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:31 smithi123 ceph-mon[44966]: purged_snaps scrub ok 2024-01-24T14:24:31.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:31 smithi123 ceph-mon[44966]: osd.11 [v2:172.21.15.194:6824/3985496260,v1:172.21.15.194:6825/3985496260] boot 2024-01-24T14:24:31.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:31 smithi123 ceph-mon[44966]: osdmap e73: 12 total, 12 up, 12 in 2024-01-24T14:24:31.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:31 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-01-24T14:24:31.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:31 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/248986388' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-24T14:24:31.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:31 smithi194 ceph-mon[45450]: purged_snaps scrub starts 2024-01-24T14:24:31.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:31 smithi194 ceph-mon[45450]: purged_snaps scrub ok 2024-01-24T14:24:31.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:31 smithi194 ceph-mon[45450]: osd.11 [v2:172.21.15.194:6824/3985496260,v1:172.21.15.194:6825/3985496260] boot 2024-01-24T14:24:31.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:31 smithi194 ceph-mon[45450]: osdmap e73: 12 total, 12 up, 12 in 2024-01-24T14:24:31.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:31 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-01-24T14:24:31.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:31 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/248986388' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-24T14:24:31.438 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:31 smithi019 ceph-mon[40663]: purged_snaps scrub starts 2024-01-24T14:24:31.438 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:31 smithi019 ceph-mon[40663]: purged_snaps scrub ok 2024-01-24T14:24:31.439 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:31 smithi019 ceph-mon[40663]: osd.11 [v2:172.21.15.194:6824/3985496260,v1:172.21.15.194:6825/3985496260] boot 2024-01-24T14:24:31.439 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:31 smithi019 ceph-mon[40663]: osdmap e73: 12 total, 12 up, 12 in 2024-01-24T14:24:31.439 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:31 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-01-24T14:24:31.439 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:31 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/248986388' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-01-24T14:24:31.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:31.995+0000 7f9a4929e700 1 -- 172.21.15.19:0/2780782244 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9a44104fa0 msgr2=0x7f9a4410cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:31.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:31.995+0000 7f9a4929e700 1 --2- 172.21.15.19:0/2780782244 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9a44104fa0 0x7f9a4410cb90 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c009f90 tx=0x7f9a3c00b330 comp rx=0 tx=0).stop 2024-01-24T14:24:31.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:31.996+0000 7f9a4929e700 1 -- 172.21.15.19:0/2780782244 shutdown_connections 2024-01-24T14:24:31.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:31.996+0000 7f9a4929e700 1 --2- 172.21.15.19:0/2780782244 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9a44104fa0 0x7f9a4410cb90 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:31.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:31.996+0000 7f9a4929e700 1 --2- 172.21.15.19:0/2780782244 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a44100110 0x7f9a441004f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:31.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:31.996+0000 7f9a4929e700 1 --2- 172.21.15.19:0/2780782244 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9a44100ac0 0x7f9a441048c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:31.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:31.996+0000 7f9a4929e700 1 -- 172.21.15.19:0/2780782244 >> 172.21.15.19:0/2780782244 conn(0x7f9a44076b10 msgr2=0x7f9a44076f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:24:31.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:31.996+0000 7f9a4929e700 1 -- 172.21.15.19:0/2780782244 shutdown_connections 2024-01-24T14:24:31.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:31.996+0000 7f9a4929e700 1 -- 172.21.15.19:0/2780782244 wait complete. 2024-01-24T14:24:31.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:31.996+0000 7f9a4929e700 1 Processor -- start 2024-01-24T14:24:32.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:31.997+0000 7f9a4929e700 1 -- start start 2024-01-24T14:24:32.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:31.997+0000 7f9a4929e700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a44100110 0x7f9a44196f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:32.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:31.997+0000 7f9a4929e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9a44100ac0 0x7f9a44197490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:32.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:31.997+0000 7f9a4929e700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9a44104fa0 0x7f9a4419b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:32.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:31.997+0000 7f9a4929e700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4419bf10 con 0x7f9a44104fa0 2024-01-24T14:24:32.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:31.997+0000 7f9a4929e700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4419c080 con 0x7f9a44100110 2024-01-24T14:24:32.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:31.997+0000 7f9a4929e700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4419c240 con 0x7f9a44100ac0 2024-01-24T14:24:32.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:31.997+0000 7f9a437fe700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9a44104fa0 0x7f9a4419b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:32.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:31.997+0000 7f9a437fe700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9a44104fa0 0x7f9a4419b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:42696/0 (socket says 172.21.15.19:42696) 2024-01-24T14:24:32.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:31.997+0000 7f9a42ffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a44100110 0x7f9a44196f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:32.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:31.997+0000 7f9a437fe700 1 -- 172.21.15.19:0/2978570410 learned_addr learned my addr 172.21.15.19:0/2978570410 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:24:32.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:31.997+0000 7f9a427fc700 1 --2- 172.21.15.19:0/2978570410 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9a44100ac0 0x7f9a44197490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:32.001 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:31.998+0000 7f9a437fe700 1 -- 172.21.15.19:0/2978570410 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9a44100ac0 msgr2=0x7f9a44197490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:32.001 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:31.998+0000 7f9a437fe700 1 --2- 172.21.15.19:0/2978570410 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9a44100ac0 0x7f9a44197490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:32.001 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:31.998+0000 7f9a437fe700 1 -- 172.21.15.19:0/2978570410 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a44100110 msgr2=0x7f9a44196f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:32.001 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:31.998+0000 7f9a437fe700 1 --2- 172.21.15.19:0/2978570410 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a44100110 0x7f9a44196f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:32.001 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:31.998+0000 7f9a437fe700 1 -- 172.21.15.19:0/2978570410 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a380095f0 con 0x7f9a44104fa0 2024-01-24T14:24:32.001 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:31.998+0000 7f9a437fe700 1 --2- 172.21.15.19:0/2978570410 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9a44104fa0 0x7f9a4419b900 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c00b6c0 tx=0x7f9a3c009510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:24:32.001 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:31.998+0000 7f9a2bfff700 1 -- 172.21.15.19:0/2978570410 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a3c00e040 con 0x7f9a44104fa0 2024-01-24T14:24:32.001 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:31.998+0000 7f9a4929e700 1 -- 172.21.15.19:0/2978570410 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a3c009c40 con 0x7f9a44104fa0 2024-01-24T14:24:32.001 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:31.998+0000 7f9a2bfff700 1 -- 172.21.15.19:0/2978570410 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9a3c01d470 con 0x7f9a44104fa0 2024-01-24T14:24:32.001 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:31.998+0000 7f9a2bfff700 1 -- 172.21.15.19:0/2978570410 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a3c0052a0 con 0x7f9a44104fa0 2024-01-24T14:24:32.001 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:31.998+0000 7f9a4929e700 1 -- 172.21.15.19:0/2978570410 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a441a3bb0 con 0x7f9a44104fa0 2024-01-24T14:24:32.002 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:31.999+0000 7f9a4929e700 1 -- 172.21.15.19:0/2978570410 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a4404ea40 con 0x7f9a44104fa0 2024-01-24T14:24:32.002 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:32.000+0000 7f9a2bfff700 1 -- 172.21.15.19:0/2978570410 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f9a3c01d5e0 con 0x7f9a44104fa0 2024-01-24T14:24:32.002 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:32.001+0000 7f9a2bfff700 1 --2- 172.21.15.19:0/2978570410 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f9a2c0a0520 0x7f9a2c0a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:32.002 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:32.001+0000 7f9a42ffd700 1 --2- 172.21.15.19:0/2978570410 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f9a2c0a0520 0x7f9a2c0a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:32.002 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:32.001+0000 7f9a2bfff700 1 -- 172.21.15.19:0/2978570410 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f9a3c0cc870 con 0x7f9a44104fa0 2024-01-24T14:24:32.003 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:32.002+0000 7f9a42ffd700 1 --2- 172.21.15.19:0/2978570410 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f9a2c0a0520 0x7f9a2c0a29e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9a34009c60 tx=0x7f9a34009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:24:32.006 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:32.005+0000 7f9a2bfff700 1 -- 172.21.15.19:0/2978570410 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f9a3c096940 con 0x7f9a44104fa0 2024-01-24T14:24:32.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:32.214+0000 7f9a4929e700 1 -- 172.21.15.19:0/2978570410 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a441982b0 con 0x7f9a44104fa0 2024-01-24T14:24:32.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:32.215+0000 7f9a2bfff700 1 -- 172.21.15.19:0/2978570410 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) v1 ==== 74+0+18459 (secure 0 0 0) 0x7f9a3c0a0020 con 0x7f9a44104fa0 2024-01-24T14:24:32.216 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:24:32.216 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":74,"fsid":"654b0c8e-bac3-11ee-95b1-87774f69a715","created":"2024-01-24T14:19:10.699369+0000","modified":"2024-01-24T14:24:31.040279+0000","last_up_change":"2024-01-24T14:24:30.032711+0000","last_in_change":"2024-01-24T14:24:13.776095+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T14:21:53.453429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"81380ec6-8e19-4eb4-8158-8fafaed8d7e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":2456786579},{"type":"v1","addr":"172.21.15.19:6803","nonce":2456786579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":2456786579},{"type":"v1","addr":"172.21.15.19:6805","nonce":2456786579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":2456786579},{"type":"v1","addr":"172.21.15.19:6809","nonce":2456786579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":2456786579},{"type":"v1","addr":"172.21.15.19:6807","nonce":2456786579}]},"public_addr":"172.21.15.19:6803/2456786579","cluster_addr":"172.21.15.19:6805/2456786579","heartbeat_back_addr":"172.21.15.19:6809/2456786579","heartbeat_front_addr":"172.21.15.19:6807/2456786579","state":["exists","up"]},{"osd":1,"uuid":"687f5036-a6d2-4aa1-86e7-7570e51ef179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":2407548889},{"type":"v1","addr":"172.21.15.123:6801","nonce":2407548889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":2407548889},{"type":"v1","addr":"172.21.15.123:6803","nonce":2407548889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":2407548889},{"type":"v1","addr":"172.21.15.123:6807","nonce":2407548889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":2407548889},{"type":"v1","addr":"172.21.15.123:6805","nonce":2407548889}]},"public_addr":"172.21.15.123:6801/2407548889","cluster_addr":"172.21.15.123:6803/2407548889","heartbeat_back_addr":"172.21.15.123:6807/2407548889","heartbeat_front_addr":"172.21.15.123:6805/2407548889","state":["exists","up"]},{"osd":2,"uuid":"3e810904-371d-4ca8-b43c-434957a2aa12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":3632698637},{"type":"v1","addr":"172.21.15.194:6801","nonce":3632698637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":3632698637},{"type":"v1","addr":"172.21.15.194:6803","nonce":3632698637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":3632698637},{"type":"v1","addr":"172.21.15.194:6807","nonce":3632698637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":3632698637},{"type":"v1","addr":"172.21.15.194:6805","nonce":3632698637}]},"public_addr":"172.21.15.194:6801/3632698637","cluster_addr":"172.21.15.194:6803/3632698637","heartbeat_back_addr":"172.21.15.194:6807/3632698637","heartbeat_front_addr":"172.21.15.194:6805/3632698637","state":["exists","up"]},{"osd":3,"uuid":"3665d6e0-010a-40bd-b0c6-682af56a0cbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":3868219159},{"type":"v1","addr":"172.21.15.19:6811","nonce":3868219159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3868219159},{"type":"v1","addr":"172.21.15.19:6813","nonce":3868219159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":3868219159},{"type":"v1","addr":"172.21.15.19:6817","nonce":3868219159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":3868219159},{"type":"v1","addr":"172.21.15.19:6815","nonce":3868219159}]},"public_addr":"172.21.15.19:6811/3868219159","cluster_addr":"172.21.15.19:6813/3868219159","heartbeat_back_addr":"172.21.15.19:6817/3868219159","heartbeat_front_addr":"172.21.15.19:6815/3868219159","state":["exists","up"]},{"osd":4,"uuid":"029fb28f-9af5-4239-8ee5-5ac5d04cd34b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":3611579809},{"type":"v1","addr":"172.21.15.123:6809","nonce":3611579809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":3611579809},{"type":"v1","addr":"172.21.15.123:6811","nonce":3611579809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":3611579809},{"type":"v1","addr":"172.21.15.123:6815","nonce":3611579809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":3611579809},{"type":"v1","addr":"172.21.15.123:6813","nonce":3611579809}]},"public_addr":"172.21.15.123:6809/3611579809","cluster_addr":"172.21.15.123:6811/3611579809","heartbeat_back_addr":"172.21.15.123:6815/3611579809","heartbeat_front_addr":"172.21.15.123:6813/3611579809","state":["exists","up"]},{"osd":5,"uuid":"6e97f23b-f803-4fed-a0a2-7aee7de4970f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":2483361871},{"type":"v1","addr":"172.21.15.194:6809","nonce":2483361871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":2483361871},{"type":"v1","addr":"172.21.15.194:6811","nonce":2483361871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":2483361871},{"type":"v1","addr":"172.21.15.194:6815","nonce":2483361871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":2483361871},{"type":"v1","addr":"172.21.15.194:6813","nonce":2483361871}]},"public_addr":"172.21.15.194:6809/2483361871","cluster_addr":"172.21.15.194:6811/2483361871","heartbeat_back_addr":"172.21.15.194:6815/2483361871","heartbeat_front_addr":"172.21.15.194:6813/2483361871","state":["exists","up"]},{"osd":6,"uuid":"c8c8ca2e-4cc6-4861-94b0-6211e4936c25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":1290513386},{"type":"v1","addr":"172.21.15.19:6819","nonce":1290513386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":1290513386},{"type":"v1","addr":"172.21.15.19:6821","nonce":1290513386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":1290513386},{"type":"v1","addr":"172.21.15.19:6825","nonce":1290513386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":1290513386},{"type":"v1","addr":"172.21.15.19:6823","nonce":1290513386}]},"public_addr":"172.21.15.19:6819/1290513386","cluster_addr":"172.21.15.19:6821/1290513386","heartbeat_back_addr":"172.21.15.19:6825/1290513386","heartbeat_front_addr":"172.21.15.19:6823/1290513386","state":["exists","up"]},{"osd":7,"uuid":"9aaea7a8-7bc3-4e32-9328-20a6c30a6d8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1391966446},{"type":"v1","addr":"172.21.15.123:6817","nonce":1391966446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1391966446},{"type":"v1","addr":"172.21.15.123:6819","nonce":1391966446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1391966446},{"type":"v1","addr":"172.21.15.123:6823","nonce":1391966446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1391966446},{"type":"v1","addr":"172.21.15.123:6821","nonce":1391966446}]},"public_addr":"172.21.15.123:6817/1391966446","cluster_addr":"172.21.15.123:6819/1391966446","heartbeat_back_addr":"172.21.15.123:6823/1391966446","heartbeat_front_addr":"172.21.15.123:6821/1391966446","state":["exists","up"]},{"osd":8,"uuid":"e918ec1b-7608-4f73-9067-29082ea63999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":2260577139},{"type":"v1","addr":"172.21.15.194:6817","nonce":2260577139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":2260577139},{"type":"v1","addr":"172.21.15.194:6819","nonce":2260577139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":2260577139},{"type":"v1","addr":"172.21.15.194:6823","nonce":2260577139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":2260577139},{"type":"v1","addr":"172.21.15.194:6821","nonce":2260577139}]},"public_addr":"172.21.15.194:6817/2260577139","cluster_addr":"172.21.15.194:6819/2260577139","heartbeat_back_addr":"172.21.15.194:6823/2260577139","heartbeat_front_addr":"172.21.15.194:6821/2260577139","state":["exists","up"]},{"osd":9,"uuid":"0fd76722-3319-4828-ab68-108607a4bd0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":4004843075},{"type":"v1","addr":"172.21.15.19:6827","nonce":4004843075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":4004843075},{"type":"v1","addr":"172.21.15.19:6829","nonce":4004843075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":4004843075},{"type":"v1","addr":"172.21.15.19:6833","nonce":4004843075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":4004843075},{"type":"v1","addr":"172.21.15.19:6831","nonce":4004843075}]},"public_addr":"172.21.15.19:6827/4004843075","cluster_addr":"172.21.15.19:6829/4004843075","heartbeat_back_addr":"172.21.15.19:6833/4004843075","heartbeat_front_addr":"172.21.15.19:6831/4004843075","state":["exists","up"]},{"osd":10,"uuid":"624bb8f5-6d72-43d5-9986-1b48b8a564ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2591845068},{"type":"v1","addr":"172.21.15.123:6825","nonce":2591845068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":2591845068},{"type":"v1","addr":"172.21.15.123:6827","nonce":2591845068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2591845068},{"type":"v1","addr":"172.21.15.123:6831","nonce":2591845068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2591845068},{"type":"v1","addr":"172.21.15.123:6829","nonce":2591845068}]},"public_addr":"172.21.15.123:6825/2591845068","cluster_addr":"172.21.15.123:6827/2591845068","heartbeat_back_addr":"172.21.15.123:6831/2591845068","heartbeat_front_addr":"172.21.15.123:6829/2591845068","state":["exists","up"]},{"osd":11,"uuid":"219a55b3-b01e-471d-90da-1a3043bf3e34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":3985496260},{"type":"v1","addr":"172.21.15.194:6825","nonce":3985496260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":3985496260},{"type":"v1","addr":"172.21.15.194:6827","nonce":3985496260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":3985496260},{"type":"v1","addr":"172.21.15.194:6831","nonce":3985496260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":3985496260},{"type":"v1","addr":"172.21.15.194:6829","nonce":3985496260}]},"public_addr":"172.21.15.194:6825/3985496260","cluster_addr":"172.21.15.194:6827/3985496260","heartbeat_back_addr":"172.21.15.194:6831/3985496260","heartbeat_front_addr":"172.21.15.194:6829/3985496260","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:21:17.278347+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:21:34.469282+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:21:51.139919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:22:09.126458+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:22:26.186801+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:22:43.178997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:23:01.109570+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:23:18.592191+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:23:35.685046+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:23:53.574820+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:24:10.987531+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:24:28.231206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.19:0/1047252857":"2024-01-25T14:20:02.911218+0000","172.21.15.19:6801/3758061162":"2024-01-25T14:20:02.911218+0000","172.21.15.19:6800/3758061162":"2024-01-25T14:20:02.911218+0000","172.21.15.19:6801/2754937713":"2024-01-25T14:19:39.853212+0000","172.21.15.19:0/4062420470":"2024-01-25T14:19:39.853212+0000","172.21.15.19:6800/2754937713":"2024-01-25T14:19:39.853212+0000","172.21.15.19:0/822613692":"2024-01-25T14:20:02.911218+0000","172.21.15.19:0/3970666946":"2024-01-25T14:19:39.853212+0000","172.21.15.19:0/1116053938":"2024-01-25T14:19:39.853212+0000","172.21.15.19:0/1456770388":"2024-01-25T14:20:02.911218+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T14:24:32.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:32.217+0000 7f9a4929e700 1 -- 172.21.15.19:0/2978570410 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f9a2c0a0520 msgr2=0x7f9a2c0a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:32.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:32.217+0000 7f9a4929e700 1 --2- 172.21.15.19:0/2978570410 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f9a2c0a0520 0x7f9a2c0a29e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9a34009c60 tx=0x7f9a34009260 comp rx=0 tx=0).stop 2024-01-24T14:24:32.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:32.217+0000 7f9a4929e700 1 -- 172.21.15.19:0/2978570410 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9a44104fa0 msgr2=0x7f9a4419b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:32.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:32.217+0000 7f9a4929e700 1 --2- 172.21.15.19:0/2978570410 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9a44104fa0 0x7f9a4419b900 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c00b6c0 tx=0x7f9a3c009510 comp rx=0 tx=0).stop 2024-01-24T14:24:32.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:32.217+0000 7f9a4929e700 1 -- 172.21.15.19:0/2978570410 shutdown_connections 2024-01-24T14:24:32.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:32.217+0000 7f9a4929e700 1 --2- 172.21.15.19:0/2978570410 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f9a2c0a0520 0x7f9a2c0a29e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:32.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:32.217+0000 7f9a4929e700 1 --2- 172.21.15.19:0/2978570410 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9a44104fa0 0x7f9a4419b900 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:32.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:32.217+0000 7f9a4929e700 1 --2- 172.21.15.19:0/2978570410 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a44100110 0x7f9a44196f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:32.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:32.217+0000 7f9a4929e700 1 --2- 172.21.15.19:0/2978570410 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9a44100ac0 0x7f9a44197490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:32.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:32.217+0000 7f9a4929e700 1 -- 172.21.15.19:0/2978570410 >> 172.21.15.19:0/2978570410 conn(0x7f9a44076b10 msgr2=0x7f9a440fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:24:32.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:32.217+0000 7f9a4929e700 1 -- 172.21.15.19:0/2978570410 shutdown_connections 2024-01-24T14:24:32.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:32.217+0000 7f9a4929e700 1 -- 172.21.15.19:0/2978570410 wait complete. 2024-01-24T14:24:32.235 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:32 smithi019 ceph-mon[40663]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-01-24T14:24:32.235 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:32 smithi019 ceph-mon[40663]: osdmap e74: 12 total, 12 up, 12 in 2024-01-24T14:24:32.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:32 smithi123 ceph-mon[44966]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-01-24T14:24:32.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:32 smithi123 ceph-mon[44966]: osdmap e74: 12 total, 12 up, 12 in 2024-01-24T14:24:32.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:32 smithi194 ceph-mon[45450]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-01-24T14:24:32.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:32 smithi194 ceph-mon[45450]: osdmap e74: 12 total, 12 up, 12 in 2024-01-24T14:24:32.609 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T14:21:53.453429+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T14:24:32.610 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-01-24T14:24:32.862 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/mon.a/config 2024-01-24T14:24:33.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:33 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/2978570410' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T14:24:33.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:33 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/2978570410' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T14:24:33.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:33 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/2978570410' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T14:24:33.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:33.897+0000 7f7b6f24e700 1 -- 172.21.15.19:0/3666502232 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7b68069220 msgr2=0x7f7b6810db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:33.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:33.897+0000 7f7b6f24e700 1 --2- 172.21.15.19:0/3666502232 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7b68069220 0x7f7b6810db80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7b6000b0a0 tx=0x7f7b6000b3b0 comp rx=0 tx=0).stop 2024-01-24T14:24:33.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:33.897+0000 7f7b6f24e700 1 -- 172.21.15.19:0/3666502232 shutdown_connections 2024-01-24T14:24:33.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:33.897+0000 7f7b6f24e700 1 --2- 172.21.15.19:0/3666502232 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7b68069220 0x7f7b6810db80 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:33.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:33.897+0000 7f7b6f24e700 1 --2- 172.21.15.19:0/3666502232 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b68106ed0 0x7f7b681072b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:33.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:33.897+0000 7f7b6f24e700 1 --2- 172.21.15.19:0/3666502232 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7b68068860 0x7f7b68068ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:33.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:33.897+0000 7f7b6f24e700 1 -- 172.21.15.19:0/3666502232 >> 172.21.15.19:0/3666502232 conn(0x7f7b68076d40 msgr2=0x7f7b68077150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:24:33.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:33.897+0000 7f7b6f24e700 1 -- 172.21.15.19:0/3666502232 shutdown_connections 2024-01-24T14:24:33.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:33.897+0000 7f7b6f24e700 1 -- 172.21.15.19:0/3666502232 wait complete. 2024-01-24T14:24:33.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:33.898+0000 7f7b6f24e700 1 Processor -- start 2024-01-24T14:24:33.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:33.898+0000 7f7b6f24e700 1 -- start start 2024-01-24T14:24:33.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:33.898+0000 7f7b6f24e700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b68068860 0x7f7b680ffcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:33.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:33.898+0000 7f7b6f24e700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7b68069220 0x7f7b68100230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:33.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:33.898+0000 7f7b6f24e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7b68106ed0 0x7f7b681007b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:33.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:33.898+0000 7f7b6f24e700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b681a53a0 con 0x7f7b68069220 2024-01-24T14:24:33.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:33.898+0000 7f7b6f24e700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b681a5510 con 0x7f7b68068860 2024-01-24T14:24:33.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:33.898+0000 7f7b6f24e700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b681a56d0 con 0x7f7b68106ed0 2024-01-24T14:24:33.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:33.898+0000 7f7b67fff700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7b68069220 0x7f7b68100230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:33.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:33.898+0000 7f7b6cfea700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b68068860 0x7f7b680ffcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:33.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:33.898+0000 7f7b67fff700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7b68069220 0x7f7b68100230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:42712/0 (socket says 172.21.15.19:42712) 2024-01-24T14:24:33.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:33.898+0000 7f7b67fff700 1 -- 172.21.15.19:0/250576545 learned_addr learned my addr 172.21.15.19:0/250576545 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:24:33.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:33.898+0000 7f7b6d7eb700 1 --2- 172.21.15.19:0/250576545 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7b68106ed0 0x7f7b681007b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:33.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:33.898+0000 7f7b67fff700 1 -- 172.21.15.19:0/250576545 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7b68106ed0 msgr2=0x7f7b681007b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:33.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:33.898+0000 7f7b67fff700 1 --2- 172.21.15.19:0/250576545 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7b68106ed0 0x7f7b681007b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:33.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:33.899+0000 7f7b67fff700 1 -- 172.21.15.19:0/250576545 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b68068860 msgr2=0x7f7b680ffcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:33.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:33.899+0000 7f7b67fff700 1 --2- 172.21.15.19:0/250576545 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b68068860 0x7f7b680ffcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:33.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:33.899+0000 7f7b67fff700 1 -- 172.21.15.19:0/250576545 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b60009d00 con 0x7f7b68069220 2024-01-24T14:24:33.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:33.899+0000 7f7b67fff700 1 --2- 172.21.15.19:0/250576545 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7b68069220 0x7f7b68100230 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c00e9a0 tx=0x7f7b5c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:24:33.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:33.899+0000 7f7b65ffb700 1 -- 172.21.15.19:0/250576545 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b5c009790 con 0x7f7b68069220 2024-01-24T14:24:33.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:33.899+0000 7f7b65ffb700 1 -- 172.21.15.19:0/250576545 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7b5c00cbd0 con 0x7f7b68069220 2024-01-24T14:24:33.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:33.899+0000 7f7b6f24e700 1 -- 172.21.15.19:0/250576545 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b681a59b0 con 0x7f7b68069220 2024-01-24T14:24:33.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:33.899+0000 7f7b65ffb700 1 -- 172.21.15.19:0/250576545 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b5c018680 con 0x7f7b68069220 2024-01-24T14:24:33.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:33.899+0000 7f7b6f24e700 1 -- 172.21.15.19:0/250576545 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b681a5f00 con 0x7f7b68069220 2024-01-24T14:24:33.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:33.900+0000 7f7b6f24e700 1 -- 172.21.15.19:0/250576545 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b6804ea40 con 0x7f7b68069220 2024-01-24T14:24:33.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:33.901+0000 7f7b65ffb700 1 -- 172.21.15.19:0/250576545 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f7b5c00cd40 con 0x7f7b68069220 2024-01-24T14:24:33.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:33.902+0000 7f7b65ffb700 1 --2- 172.21.15.19:0/250576545 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f7b500a05a0 0x7f7b500a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:33.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:33.902+0000 7f7b65ffb700 1 -- 172.21.15.19:0/250576545 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f7b5c014070 con 0x7f7b68069220 2024-01-24T14:24:33.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:33.902+0000 7f7b6cfea700 1 --2- 172.21.15.19:0/250576545 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f7b500a05a0 0x7f7b500a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:33.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:33.903+0000 7f7b6cfea700 1 --2- 172.21.15.19:0/250576545 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f7b500a05a0 0x7f7b500a2a60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7b58005950 tx=0x7f7b5800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:24:33.907 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:33.906+0000 7f7b65ffb700 1 -- 172.21.15.19:0/250576545 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f7b5c052120 con 0x7f7b68069220 2024-01-24T14:24:34.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:34.113+0000 7f7b6f24e700 1 -- 172.21.15.19:0/250576545 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7b681a6210 con 0x7f7b68069220 2024-01-24T14:24:34.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:34.113+0000 7f7b65ffb700 1 -- 172.21.15.19:0/250576545 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v74) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7b5c095720 con 0x7f7b68069220 2024-01-24T14:24:34.114 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 1 2024-01-24T14:24:34.117 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:34.116+0000 7f7b6f24e700 1 -- 172.21.15.19:0/250576545 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f7b500a05a0 msgr2=0x7f7b500a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:34.117 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:34.116+0000 7f7b6f24e700 1 --2- 172.21.15.19:0/250576545 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f7b500a05a0 0x7f7b500a2a60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7b58005950 tx=0x7f7b5800b410 comp rx=0 tx=0).stop 2024-01-24T14:24:34.117 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:34.116+0000 7f7b6f24e700 1 -- 172.21.15.19:0/250576545 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7b68069220 msgr2=0x7f7b68100230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:34.117 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:34.116+0000 7f7b6f24e700 1 --2- 172.21.15.19:0/250576545 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7b68069220 0x7f7b68100230 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c00e9a0 tx=0x7f7b5c00ecb0 comp rx=0 tx=0).stop 2024-01-24T14:24:34.117 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:34.116+0000 7f7b6f24e700 1 -- 172.21.15.19:0/250576545 shutdown_connections 2024-01-24T14:24:34.117 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:34.116+0000 7f7b6f24e700 1 --2- 172.21.15.19:0/250576545 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7b68106ed0 0x7f7b681007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:34.117 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:34.116+0000 7f7b6f24e700 1 --2- 172.21.15.19:0/250576545 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b68068860 0x7f7b680ffcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:34.117 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:34.116+0000 7f7b6f24e700 1 --2- 172.21.15.19:0/250576545 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f7b500a05a0 0x7f7b500a2a60 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:34.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:34.116+0000 7f7b6f24e700 1 --2- 172.21.15.19:0/250576545 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7b68069220 0x7f7b68100230 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:34.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:34.116+0000 7f7b6f24e700 1 -- 172.21.15.19:0/250576545 >> 172.21.15.19:0/250576545 conn(0x7f7b68076d40 msgr2=0x7f7b680fe770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:24:34.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:34.116+0000 7f7b6f24e700 1 -- 172.21.15.19:0/250576545 shutdown_connections 2024-01-24T14:24:34.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:34.116+0000 7f7b6f24e700 1 -- 172.21.15.19:0/250576545 wait complete. 2024-01-24T14:24:34.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:34 smithi194 ceph-mon[45450]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:24:34.435 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:34 smithi019 ceph-mon[40663]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:24:34.493 INFO:tasks.cephadm:Adding mds.a on smithi019 2024-01-24T14:24:34.494 INFO:tasks.cephadm:Adding mds.d on smithi019 2024-01-24T14:24:34.494 INFO:tasks.cephadm:Adding mds.g on smithi019 2024-01-24T14:24:34.494 INFO:tasks.cephadm:Adding mds.j on smithi019 2024-01-24T14:24:34.494 INFO:tasks.cephadm:Adding mds.b on smithi123 2024-01-24T14:24:34.494 INFO:tasks.cephadm:Adding mds.e on smithi123 2024-01-24T14:24:34.494 INFO:tasks.cephadm:Adding mds.h on smithi123 2024-01-24T14:24:34.494 INFO:tasks.cephadm:Adding mds.k on smithi123 2024-01-24T14:24:34.494 INFO:tasks.cephadm:Adding mds.c on smithi194 2024-01-24T14:24:34.494 INFO:tasks.cephadm:Adding mds.f on smithi194 2024-01-24T14:24:34.494 INFO:tasks.cephadm:Adding mds.i on smithi194 2024-01-24T14:24:34.494 INFO:tasks.cephadm:Adding mds.l on smithi194 2024-01-24T14:24:34.494 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 -- ceph orch apply mds all '12;smithi019=a;smithi019=d;smithi019=g;smithi019=j;smithi123=b;smithi123=e;smithi123=h;smithi123=k;smithi194=c;smithi194=f;smithi194=i;smithi194=l' 2024-01-24T14:24:34.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:34 smithi123 ceph-mon[44966]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:24:34.728 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/mon.c/config 2024-01-24T14:24:35.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:35 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/250576545' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-01-24T14:24:35.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:35 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/250576545' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-01-24T14:24:35.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:35 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/250576545' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-01-24T14:24:35.704 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.701+0000 7fc53696e700 1 -- 172.21.15.194:0/4156826278 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc530106ac0 msgr2=0x7fc530108f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:35.704 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.701+0000 7fc53696e700 1 --2- 172.21.15.194:0/4156826278 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc530106ac0 0x7fc530108f30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc52800b0a0 tx=0x7fc52800b3b0 comp rx=0 tx=0).stop 2024-01-24T14:24:35.704 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.701+0000 7fc53696e700 1 -- 172.21.15.194:0/4156826278 shutdown_connections 2024-01-24T14:24:35.704 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.701+0000 7fc53696e700 1 --2- 172.21.15.194:0/4156826278 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc530106ac0 0x7fc530108f30 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:35.704 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.701+0000 7fc53696e700 1 --2- 172.21.15.194:0/4156826278 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc530101860 0x7fc530103c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:35.704 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.701+0000 7fc53696e700 1 --2- 172.21.15.194:0/4156826278 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc530104190 0x7fc530106580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:35.704 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.701+0000 7fc53696e700 1 -- 172.21.15.194:0/4156826278 >> 172.21.15.194:0/4156826278 conn(0x7fc5300fb250 msgr2=0x7fc5300fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:24:35.704 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.701+0000 7fc53696e700 1 -- 172.21.15.194:0/4156826278 shutdown_connections 2024-01-24T14:24:35.705 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.701+0000 7fc53696e700 1 -- 172.21.15.194:0/4156826278 wait complete. 2024-01-24T14:24:35.705 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.701+0000 7fc53696e700 1 Processor -- start 2024-01-24T14:24:35.705 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.701+0000 7fc53696e700 1 -- start start 2024-01-24T14:24:35.705 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.702+0000 7fc53696e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc530101860 0x7fc530198ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:35.705 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.702+0000 7fc53696e700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc530104190 0x7fc530199420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:35.705 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.702+0000 7fc53696e700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc530106ac0 0x7fc5301a04a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:35.705 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.702+0000 7fc53696e700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5301a0a70 con 0x7fc530106ac0 2024-01-24T14:24:35.705 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.702+0000 7fc53696e700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5301a0bb0 con 0x7fc530104190 2024-01-24T14:24:35.705 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.702+0000 7fc53696e700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5301a0d40 con 0x7fc530101860 2024-01-24T14:24:35.705 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.702+0000 7fc52ffff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc530101860 0x7fc530198ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:35.705 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.702+0000 7fc52ffff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc530101860 0x7fc530198ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:58318/0 (socket says 172.21.15.194:58318) 2024-01-24T14:24:35.705 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.702+0000 7fc52ffff700 1 -- 172.21.15.194:0/3833928790 learned_addr learned my addr 172.21.15.194:0/3833928790 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-24T14:24:35.705 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.702+0000 7fc52f7fe700 1 --2- 172.21.15.194:0/3833928790 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc530104190 0x7fc530199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:35.706 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.702+0000 7fc52ffff700 1 -- 172.21.15.194:0/3833928790 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc530104190 msgr2=0x7fc530199420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:35.706 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.702+0000 7fc52ffff700 1 --2- 172.21.15.194:0/3833928790 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc530104190 0x7fc530199420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:35.706 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.702+0000 7fc52ffff700 1 -- 172.21.15.194:0/3833928790 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc530106ac0 msgr2=0x7fc5301a04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:35.706 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.702+0000 7fc534f0b700 1 --2- 172.21.15.194:0/3833928790 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc530106ac0 0x7fc5301a04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:35.706 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.702+0000 7fc52ffff700 1 --2- 172.21.15.194:0/3833928790 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc530106ac0 0x7fc5301a04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:35.706 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.702+0000 7fc52ffff700 1 -- 172.21.15.194:0/3833928790 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc528009d00 con 0x7fc530101860 2024-01-24T14:24:35.706 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.702+0000 7fc52ffff700 1 --2- 172.21.15.194:0/3833928790 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc530101860 0x7fc530198ee0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc52400b560 tx=0x7fc52400b920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:24:35.706 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.702+0000 7fc52d7fa700 1 -- 172.21.15.194:0/3833928790 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc524011830 con 0x7fc530101860 2024-01-24T14:24:35.706 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.702+0000 7fc52d7fa700 1 -- 172.21.15.194:0/3833928790 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc524011e50 con 0x7fc530101860 2024-01-24T14:24:35.706 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.702+0000 7fc52d7fa700 1 -- 172.21.15.194:0/3833928790 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52400f2e0 con 0x7fc530101860 2024-01-24T14:24:35.706 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.702+0000 7fc53696e700 1 -- 172.21.15.194:0/3833928790 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5301a0fc0 con 0x7fc530101860 2024-01-24T14:24:35.706 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.702+0000 7fc53696e700 1 -- 172.21.15.194:0/3833928790 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5301a14b0 con 0x7fc530101860 2024-01-24T14:24:35.707 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.703+0000 7fc53696e700 1 -- 172.21.15.194:0/3833928790 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc530108640 con 0x7fc530101860 2024-01-24T14:24:35.707 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.704+0000 7fc52d7fa700 1 -- 172.21.15.194:0/3833928790 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7fc524011990 con 0x7fc530101860 2024-01-24T14:24:35.707 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.705+0000 7fc52d7fa700 1 --2- 172.21.15.194:0/3833928790 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fc5180a05a0 0x7fc5180a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:35.708 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.705+0000 7fc52d7fa700 1 -- 172.21.15.194:0/3833928790 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc5240caf90 con 0x7fc530101860 2024-01-24T14:24:35.708 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.705+0000 7fc52f7fe700 1 --2- 172.21.15.194:0/3833928790 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fc5180a05a0 0x7fc5180a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:35.708 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.706+0000 7fc52f7fe700 1 --2- 172.21.15.194:0/3833928790 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fc5180a05a0 0x7fc5180a2a60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc520006fd0 tx=0x7fc520008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:24:35.710 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.709+0000 7fc52d7fa700 1 -- 172.21.15.194:0/3833928790 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fc5240fe120 con 0x7fc530101860 2024-01-24T14:24:35.909 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.908+0000 7fc53696e700 1 -- 172.21.15.194:0/3833928790 --> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi019=a;smithi019=d;smithi019=g;smithi019=j;smithi123=b;smithi123=e;smithi123=h;smithi123=k;smithi194=c;smithi194=f;smithi194=i;smithi194=l", "target": ["mon-mgr", ""]}) v1 -- 0x7fc530061380 con 0x7fc5180a05a0 2024-01-24T14:24:35.919 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.918+0000 7fc52d7fa700 1 -- 172.21.15.194:0/3833928790 <== mgr.14152 v2:172.21.15.19:6800/1634350318 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fc530061380 con 0x7fc5180a05a0 2024-01-24T14:24:35.920 INFO:teuthology.orchestra.run.smithi194.stdout:Scheduled mds.all update... 2024-01-24T14:24:35.922 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.920+0000 7fc53696e700 1 -- 172.21.15.194:0/3833928790 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fc5180a05a0 msgr2=0x7fc5180a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:35.922 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.920+0000 7fc53696e700 1 --2- 172.21.15.194:0/3833928790 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fc5180a05a0 0x7fc5180a2a60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc520006fd0 tx=0x7fc520008040 comp rx=0 tx=0).stop 2024-01-24T14:24:35.922 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.920+0000 7fc53696e700 1 -- 172.21.15.194:0/3833928790 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc530101860 msgr2=0x7fc530198ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:35.922 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.920+0000 7fc53696e700 1 --2- 172.21.15.194:0/3833928790 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc530101860 0x7fc530198ee0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc52400b560 tx=0x7fc52400b920 comp rx=0 tx=0).stop 2024-01-24T14:24:35.922 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.920+0000 7fc53696e700 1 -- 172.21.15.194:0/3833928790 shutdown_connections 2024-01-24T14:24:35.922 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.920+0000 7fc53696e700 1 --2- 172.21.15.194:0/3833928790 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fc5180a05a0 0x7fc5180a2a60 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:35.922 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.920+0000 7fc53696e700 1 --2- 172.21.15.194:0/3833928790 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc530106ac0 0x7fc5301a04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:35.922 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.920+0000 7fc53696e700 1 --2- 172.21.15.194:0/3833928790 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc530101860 0x7fc530198ee0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:35.922 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.920+0000 7fc53696e700 1 --2- 172.21.15.194:0/3833928790 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc530104190 0x7fc530199420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:35.922 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.920+0000 7fc53696e700 1 -- 172.21.15.194:0/3833928790 >> 172.21.15.194:0/3833928790 conn(0x7fc5300fb250 msgr2=0x7fc5300ffee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:24:35.922 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.920+0000 7fc53696e700 1 -- 172.21.15.194:0/3833928790 shutdown_connections 2024-01-24T14:24:35.922 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-24T14:24:35.920+0000 7fc53696e700 1 -- 172.21.15.194:0/3833928790 wait complete. 2024-01-24T14:24:36.286 DEBUG:teuthology.orchestra.run.smithi019:mds.a> sudo journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mds.a.service 2024-01-24T14:24:36.289 DEBUG:teuthology.orchestra.run.smithi019:mds.d> sudo journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mds.d.service 2024-01-24T14:24:36.324 INFO:journalctl@ceph.mds.a.smithi019.stdout:-- Logs begin at Wed 2024-01-24 14:03:52 UTC. -- 2024-01-24T14:24:36.325 DEBUG:teuthology.orchestra.run.smithi019:mds.g> sudo journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mds.g.service 2024-01-24T14:24:36.355 INFO:journalctl@ceph.mds.d.smithi019.stdout:-- Logs begin at Wed 2024-01-24 14:03:52 UTC. -- 2024-01-24T14:24:36.361 DEBUG:teuthology.orchestra.run.smithi019:mds.j> sudo journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mds.j.service 2024-01-24T14:24:36.388 INFO:journalctl@ceph.mds.g.smithi019.stdout:-- Logs begin at Wed 2024-01-24 14:03:52 UTC. -- 2024-01-24T14:24:36.389 DEBUG:teuthology.orchestra.run.smithi123:mds.b> sudo journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mds.b.service 2024-01-24T14:24:36.396 DEBUG:teuthology.orchestra.run.smithi123:mds.e> sudo journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mds.e.service 2024-01-24T14:24:36.420 INFO:journalctl@ceph.mds.j.smithi019.stdout:-- Logs begin at Wed 2024-01-24 14:03:52 UTC. -- 2024-01-24T14:24:36.430 INFO:journalctl@ceph.mds.b.smithi123.stdout:-- Logs begin at Wed 2024-01-24 14:03:45 UTC. -- 2024-01-24T14:24:36.431 DEBUG:teuthology.orchestra.run.smithi123:mds.h> sudo journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mds.h.service 2024-01-24T14:24:36.461 INFO:journalctl@ceph.mds.e.smithi123.stdout:-- Logs begin at Wed 2024-01-24 14:03:45 UTC. -- 2024-01-24T14:24:36.462 DEBUG:teuthology.orchestra.run.smithi123:mds.k> sudo journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mds.k.service 2024-01-24T14:24:36.489 INFO:journalctl@ceph.mds.h.smithi123.stdout:-- Logs begin at Wed 2024-01-24 14:03:45 UTC. -- 2024-01-24T14:24:36.490 DEBUG:teuthology.orchestra.run.smithi194:mds.c> sudo journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mds.c.service 2024-01-24T14:24:36.493 DEBUG:teuthology.orchestra.run.smithi194:mds.f> sudo journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mds.f.service 2024-01-24T14:24:36.517 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:36 smithi194 ceph-mon[45450]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:24:36.517 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:36 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:36.517 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:36 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:24:36.517 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:36 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:36.517 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:36 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:24:36.517 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:36 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:36.517 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:36 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T14:24:36.517 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:36 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T14:24:36.517 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:36 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:36.518 INFO:journalctl@ceph.mds.k.smithi123.stdout:-- Logs begin at Wed 2024-01-24 14:03:45 UTC. -- 2024-01-24T14:24:36.518 DEBUG:teuthology.orchestra.run.smithi194:mds.i> sudo journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mds.i.service 2024-01-24T14:24:36.523 INFO:journalctl@ceph.mds.c.smithi194.stdout:-- Logs begin at Wed 2024-01-24 14:04:01 UTC. -- 2024-01-24T14:24:36.525 DEBUG:teuthology.orchestra.run.smithi194:mds.l> sudo journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mds.l.service 2024-01-24T14:24:36.544 INFO:journalctl@ceph.mds.f.smithi194.stdout:-- Logs begin at Wed 2024-01-24 14:04:01 UTC. -- 2024-01-24T14:24:36.545 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-01-24T14:24:36.545 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T14:24:36.550 INFO:journalctl@ceph.mds.i.smithi194.stdout:-- Logs begin at Wed 2024-01-24 14:04:01 UTC. -- 2024-01-24T14:24:36.571 INFO:journalctl@ceph.mds.l.smithi194.stdout:-- Logs begin at Wed 2024-01-24 14:04:01 UTC. -- 2024-01-24T14:24:36.575 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:36 smithi019 ceph-mon[40663]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:24:36.575 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:36 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:36.575 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:36 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:24:36.575 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:36 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:36.575 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:36 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:24:36.575 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:36 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:36.575 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:36 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T14:24:36.575 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:36 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T14:24:36.575 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:36 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:36.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.709+0000 7f733b86f700 1 -- 172.21.15.19:0/759861526 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7334100900 msgr2=0x7f7334100d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:36.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.709+0000 7f733b86f700 1 --2- 172.21.15.19:0/759861526 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7334100900 0x7f7334100d80 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f732c00b0a0 tx=0x7f732c00b3b0 comp rx=0 tx=0).stop 2024-01-24T14:24:36.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.709+0000 7f733b86f700 1 -- 172.21.15.19:0/759861526 shutdown_connections 2024-01-24T14:24:36.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.709+0000 7f733b86f700 1 --2- 172.21.15.19:0/759861526 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7334100900 0x7f7334100d80 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:36.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.709+0000 7f733b86f700 1 --2- 172.21.15.19:0/759861526 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73340ff5e0 0x7f73340ffa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:36.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.710+0000 7f733b86f700 1 --2- 172.21.15.19:0/759861526 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f73340fff40 0x7f73341003c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:36.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.710+0000 7f733b86f700 1 -- 172.21.15.19:0/759861526 >> 172.21.15.19:0/759861526 conn(0x7f73340fb200 msgr2=0x7f73340fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:24:36.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.710+0000 7f733b86f700 1 -- 172.21.15.19:0/759861526 shutdown_connections 2024-01-24T14:24:36.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.710+0000 7f733b86f700 1 -- 172.21.15.19:0/759861526 wait complete. 2024-01-24T14:24:36.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.710+0000 7f733b86f700 1 Processor -- start 2024-01-24T14:24:36.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.710+0000 7f733b86f700 1 -- start start 2024-01-24T14:24:36.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.710+0000 7f733b86f700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f73340ff5e0 0x7f7334198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:36.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.711+0000 7f733b86f700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73340fff40 0x7f73341991b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:36.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.711+0000 7f733b86f700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7334100900 0x7f73341a01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:36.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.711+0000 7f733b86f700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73341a07b0 con 0x7f73340ff5e0 2024-01-24T14:24:36.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.711+0000 7f733b86f700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73341a0920 con 0x7f73340fff40 2024-01-24T14:24:36.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.711+0000 7f733b86f700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73341a0ae0 con 0x7f7334100900 2024-01-24T14:24:36.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.711+0000 7f733960b700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f73340ff5e0 0x7f7334198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:36.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.711+0000 7f733960b700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f73340ff5e0 0x7f7334198c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:42736/0 (socket says 172.21.15.19:42736) 2024-01-24T14:24:36.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.711+0000 7f733960b700 1 -- 172.21.15.19:0/3921759920 learned_addr learned my addr 172.21.15.19:0/3921759920 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:24:36.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.711+0000 7f7339e0c700 1 --2- 172.21.15.19:0/3921759920 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7334100900 0x7f73341a01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:36.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.711+0000 7f7338e0a700 1 --2- 172.21.15.19:0/3921759920 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73340fff40 0x7f73341991b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:36.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.711+0000 7f733960b700 1 -- 172.21.15.19:0/3921759920 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7334100900 msgr2=0x7f73341a01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:36.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.711+0000 7f733960b700 1 --2- 172.21.15.19:0/3921759920 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7334100900 0x7f73341a01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:36.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.711+0000 7f733960b700 1 -- 172.21.15.19:0/3921759920 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73340fff40 msgr2=0x7f73341991b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:36.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.711+0000 7f733960b700 1 --2- 172.21.15.19:0/3921759920 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73340fff40 0x7f73341991b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:36.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.711+0000 7f733960b700 1 -- 172.21.15.19:0/3921759920 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f732c009d00 con 0x7f73340ff5e0 2024-01-24T14:24:36.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.711+0000 7f733960b700 1 --2- 172.21.15.19:0/3921759920 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f73340ff5e0 0x7f7334198c70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f732800e9a0 tx=0x7f732800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:24:36.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.711+0000 7f73267fc700 1 -- 172.21.15.19:0/3921759920 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7328009790 con 0x7f73340ff5e0 2024-01-24T14:24:36.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.711+0000 7f73267fc700 1 -- 172.21.15.19:0/3921759920 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f732800cbd0 con 0x7f73340ff5e0 2024-01-24T14:24:36.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.711+0000 7f733b86f700 1 -- 172.21.15.19:0/3921759920 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73341a0dc0 con 0x7f73340ff5e0 2024-01-24T14:24:36.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.711+0000 7f73267fc700 1 -- 172.21.15.19:0/3921759920 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7328018680 con 0x7f73340ff5e0 2024-01-24T14:24:36.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.711+0000 7f733b86f700 1 -- 172.21.15.19:0/3921759920 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73341a1240 con 0x7f73340ff5e0 2024-01-24T14:24:36.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.712+0000 7f733b86f700 1 -- 172.21.15.19:0/3921759920 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f733410a3c0 con 0x7f73340ff5e0 2024-01-24T14:24:36.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.713+0000 7f73267fc700 1 -- 172.21.15.19:0/3921759920 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f732800cd40 con 0x7f73340ff5e0 2024-01-24T14:24:36.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.714+0000 7f73267fc700 1 --2- 172.21.15.19:0/3921759920 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f73200a05f0 0x7f73200a2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:36.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.714+0000 7f7338e0a700 1 --2- 172.21.15.19:0/3921759920 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f73200a05f0 0x7f73200a2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:36.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.714+0000 7f73267fc700 1 -- 172.21.15.19:0/3921759920 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f7328014070 con 0x7f73340ff5e0 2024-01-24T14:24:36.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.715+0000 7f7338e0a700 1 --2- 172.21.15.19:0/3921759920 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f73200a05f0 0x7f73200a2ab0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7330005fd0 tx=0x7f7330005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:24:36.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.718+0000 7f73267fc700 1 -- 172.21.15.19:0/3921759920 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f73280959f0 con 0x7f73340ff5e0 2024-01-24T14:24:36.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:36 smithi123 ceph-mon[44966]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:24:36.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:36 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:36.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:36 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:24:36.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:36 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:36.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:36 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:24:36.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:36 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:36.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:36 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T14:24:36.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:36 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T14:24:36.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:36 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:36.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.932+0000 7f733b86f700 1 -- 172.21.15.19:0/3921759920 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f733402ce40 con 0x7f73340ff5e0 2024-01-24T14:24:36.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.933+0000 7f73267fc700 1 -- 172.21.15.19:0/3921759920 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) v1 ==== 74+0+18459 (secure 0 0 0) 0x7f7328099010 con 0x7f73340ff5e0 2024-01-24T14:24:36.934 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:24:36.935 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":74,"fsid":"654b0c8e-bac3-11ee-95b1-87774f69a715","created":"2024-01-24T14:19:10.699369+0000","modified":"2024-01-24T14:24:31.040279+0000","last_up_change":"2024-01-24T14:24:30.032711+0000","last_in_change":"2024-01-24T14:24:13.776095+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T14:21:53.453429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"81380ec6-8e19-4eb4-8158-8fafaed8d7e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":2456786579},{"type":"v1","addr":"172.21.15.19:6803","nonce":2456786579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":2456786579},{"type":"v1","addr":"172.21.15.19:6805","nonce":2456786579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":2456786579},{"type":"v1","addr":"172.21.15.19:6809","nonce":2456786579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":2456786579},{"type":"v1","addr":"172.21.15.19:6807","nonce":2456786579}]},"public_addr":"172.21.15.19:6803/2456786579","cluster_addr":"172.21.15.19:6805/2456786579","heartbeat_back_addr":"172.21.15.19:6809/2456786579","heartbeat_front_addr":"172.21.15.19:6807/2456786579","state":["exists","up"]},{"osd":1,"uuid":"687f5036-a6d2-4aa1-86e7-7570e51ef179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":2407548889},{"type":"v1","addr":"172.21.15.123:6801","nonce":2407548889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":2407548889},{"type":"v1","addr":"172.21.15.123:6803","nonce":2407548889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":2407548889},{"type":"v1","addr":"172.21.15.123:6807","nonce":2407548889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":2407548889},{"type":"v1","addr":"172.21.15.123:6805","nonce":2407548889}]},"public_addr":"172.21.15.123:6801/2407548889","cluster_addr":"172.21.15.123:6803/2407548889","heartbeat_back_addr":"172.21.15.123:6807/2407548889","heartbeat_front_addr":"172.21.15.123:6805/2407548889","state":["exists","up"]},{"osd":2,"uuid":"3e810904-371d-4ca8-b43c-434957a2aa12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":3632698637},{"type":"v1","addr":"172.21.15.194:6801","nonce":3632698637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":3632698637},{"type":"v1","addr":"172.21.15.194:6803","nonce":3632698637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":3632698637},{"type":"v1","addr":"172.21.15.194:6807","nonce":3632698637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":3632698637},{"type":"v1","addr":"172.21.15.194:6805","nonce":3632698637}]},"public_addr":"172.21.15.194:6801/3632698637","cluster_addr":"172.21.15.194:6803/3632698637","heartbeat_back_addr":"172.21.15.194:6807/3632698637","heartbeat_front_addr":"172.21.15.194:6805/3632698637","state":["exists","up"]},{"osd":3,"uuid":"3665d6e0-010a-40bd-b0c6-682af56a0cbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":3868219159},{"type":"v1","addr":"172.21.15.19:6811","nonce":3868219159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3868219159},{"type":"v1","addr":"172.21.15.19:6813","nonce":3868219159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":3868219159},{"type":"v1","addr":"172.21.15.19:6817","nonce":3868219159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":3868219159},{"type":"v1","addr":"172.21.15.19:6815","nonce":3868219159}]},"public_addr":"172.21.15.19:6811/3868219159","cluster_addr":"172.21.15.19:6813/3868219159","heartbeat_back_addr":"172.21.15.19:6817/3868219159","heartbeat_front_addr":"172.21.15.19:6815/3868219159","state":["exists","up"]},{"osd":4,"uuid":"029fb28f-9af5-4239-8ee5-5ac5d04cd34b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":3611579809},{"type":"v1","addr":"172.21.15.123:6809","nonce":3611579809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":3611579809},{"type":"v1","addr":"172.21.15.123:6811","nonce":3611579809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":3611579809},{"type":"v1","addr":"172.21.15.123:6815","nonce":3611579809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":3611579809},{"type":"v1","addr":"172.21.15.123:6813","nonce":3611579809}]},"public_addr":"172.21.15.123:6809/3611579809","cluster_addr":"172.21.15.123:6811/3611579809","heartbeat_back_addr":"172.21.15.123:6815/3611579809","heartbeat_front_addr":"172.21.15.123:6813/3611579809","state":["exists","up"]},{"osd":5,"uuid":"6e97f23b-f803-4fed-a0a2-7aee7de4970f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":2483361871},{"type":"v1","addr":"172.21.15.194:6809","nonce":2483361871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":2483361871},{"type":"v1","addr":"172.21.15.194:6811","nonce":2483361871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":2483361871},{"type":"v1","addr":"172.21.15.194:6815","nonce":2483361871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":2483361871},{"type":"v1","addr":"172.21.15.194:6813","nonce":2483361871}]},"public_addr":"172.21.15.194:6809/2483361871","cluster_addr":"172.21.15.194:6811/2483361871","heartbeat_back_addr":"172.21.15.194:6815/2483361871","heartbeat_front_addr":"172.21.15.194:6813/2483361871","state":["exists","up"]},{"osd":6,"uuid":"c8c8ca2e-4cc6-4861-94b0-6211e4936c25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":1290513386},{"type":"v1","addr":"172.21.15.19:6819","nonce":1290513386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":1290513386},{"type":"v1","addr":"172.21.15.19:6821","nonce":1290513386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":1290513386},{"type":"v1","addr":"172.21.15.19:6825","nonce":1290513386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":1290513386},{"type":"v1","addr":"172.21.15.19:6823","nonce":1290513386}]},"public_addr":"172.21.15.19:6819/1290513386","cluster_addr":"172.21.15.19:6821/1290513386","heartbeat_back_addr":"172.21.15.19:6825/1290513386","heartbeat_front_addr":"172.21.15.19:6823/1290513386","state":["exists","up"]},{"osd":7,"uuid":"9aaea7a8-7bc3-4e32-9328-20a6c30a6d8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1391966446},{"type":"v1","addr":"172.21.15.123:6817","nonce":1391966446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1391966446},{"type":"v1","addr":"172.21.15.123:6819","nonce":1391966446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1391966446},{"type":"v1","addr":"172.21.15.123:6823","nonce":1391966446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1391966446},{"type":"v1","addr":"172.21.15.123:6821","nonce":1391966446}]},"public_addr":"172.21.15.123:6817/1391966446","cluster_addr":"172.21.15.123:6819/1391966446","heartbeat_back_addr":"172.21.15.123:6823/1391966446","heartbeat_front_addr":"172.21.15.123:6821/1391966446","state":["exists","up"]},{"osd":8,"uuid":"e918ec1b-7608-4f73-9067-29082ea63999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":2260577139},{"type":"v1","addr":"172.21.15.194:6817","nonce":2260577139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":2260577139},{"type":"v1","addr":"172.21.15.194:6819","nonce":2260577139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":2260577139},{"type":"v1","addr":"172.21.15.194:6823","nonce":2260577139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":2260577139},{"type":"v1","addr":"172.21.15.194:6821","nonce":2260577139}]},"public_addr":"172.21.15.194:6817/2260577139","cluster_addr":"172.21.15.194:6819/2260577139","heartbeat_back_addr":"172.21.15.194:6823/2260577139","heartbeat_front_addr":"172.21.15.194:6821/2260577139","state":["exists","up"]},{"osd":9,"uuid":"0fd76722-3319-4828-ab68-108607a4bd0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":4004843075},{"type":"v1","addr":"172.21.15.19:6827","nonce":4004843075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":4004843075},{"type":"v1","addr":"172.21.15.19:6829","nonce":4004843075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":4004843075},{"type":"v1","addr":"172.21.15.19:6833","nonce":4004843075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":4004843075},{"type":"v1","addr":"172.21.15.19:6831","nonce":4004843075}]},"public_addr":"172.21.15.19:6827/4004843075","cluster_addr":"172.21.15.19:6829/4004843075","heartbeat_back_addr":"172.21.15.19:6833/4004843075","heartbeat_front_addr":"172.21.15.19:6831/4004843075","state":["exists","up"]},{"osd":10,"uuid":"624bb8f5-6d72-43d5-9986-1b48b8a564ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2591845068},{"type":"v1","addr":"172.21.15.123:6825","nonce":2591845068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":2591845068},{"type":"v1","addr":"172.21.15.123:6827","nonce":2591845068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2591845068},{"type":"v1","addr":"172.21.15.123:6831","nonce":2591845068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2591845068},{"type":"v1","addr":"172.21.15.123:6829","nonce":2591845068}]},"public_addr":"172.21.15.123:6825/2591845068","cluster_addr":"172.21.15.123:6827/2591845068","heartbeat_back_addr":"172.21.15.123:6831/2591845068","heartbeat_front_addr":"172.21.15.123:6829/2591845068","state":["exists","up"]},{"osd":11,"uuid":"219a55b3-b01e-471d-90da-1a3043bf3e34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":3985496260},{"type":"v1","addr":"172.21.15.194:6825","nonce":3985496260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":3985496260},{"type":"v1","addr":"172.21.15.194:6827","nonce":3985496260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":3985496260},{"type":"v1","addr":"172.21.15.194:6831","nonce":3985496260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":3985496260},{"type":"v1","addr":"172.21.15.194:6829","nonce":3985496260}]},"public_addr":"172.21.15.194:6825/3985496260","cluster_addr":"172.21.15.194:6827/3985496260","heartbeat_back_addr":"172.21.15.194:6831/3985496260","heartbeat_front_addr":"172.21.15.194:6829/3985496260","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:21:17.278347+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:21:34.469282+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:21:51.139919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:22:09.126458+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:22:26.186801+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:22:43.178997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:23:01.109570+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:23:18.592191+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:23:35.685046+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:23:53.574820+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:24:10.987531+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:24:28.231206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.19:0/1047252857":"2024-01-25T14:20:02.911218+0000","172.21.15.19:6801/3758061162":"2024-01-25T14:20:02.911218+0000","172.21.15.19:6800/3758061162":"2024-01-25T14:20:02.911218+0000","172.21.15.19:6801/2754937713":"2024-01-25T14:19:39.853212+0000","172.21.15.19:0/4062420470":"2024-01-25T14:19:39.853212+0000","172.21.15.19:6800/2754937713":"2024-01-25T14:19:39.853212+0000","172.21.15.19:0/822613692":"2024-01-25T14:20:02.911218+0000","172.21.15.19:0/3970666946":"2024-01-25T14:19:39.853212+0000","172.21.15.19:0/1116053938":"2024-01-25T14:19:39.853212+0000","172.21.15.19:0/1456770388":"2024-01-25T14:20:02.911218+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T14:24:36.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.935+0000 7f733b86f700 1 -- 172.21.15.19:0/3921759920 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f73200a05f0 msgr2=0x7f73200a2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:36.937 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.935+0000 7f733b86f700 1 --2- 172.21.15.19:0/3921759920 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f73200a05f0 0x7f73200a2ab0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7330005fd0 tx=0x7f7330005f60 comp rx=0 tx=0).stop 2024-01-24T14:24:36.937 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.935+0000 7f733b86f700 1 -- 172.21.15.19:0/3921759920 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f73340ff5e0 msgr2=0x7f7334198c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:36.937 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.935+0000 7f733b86f700 1 --2- 172.21.15.19:0/3921759920 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f73340ff5e0 0x7f7334198c70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f732800e9a0 tx=0x7f732800ecb0 comp rx=0 tx=0).stop 2024-01-24T14:24:36.937 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.935+0000 7f733b86f700 1 -- 172.21.15.19:0/3921759920 shutdown_connections 2024-01-24T14:24:36.937 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.935+0000 7f733b86f700 1 --2- 172.21.15.19:0/3921759920 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f73200a05f0 0x7f73200a2ab0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:36.937 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.935+0000 7f733b86f700 1 --2- 172.21.15.19:0/3921759920 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f73340ff5e0 0x7f7334198c70 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:36.937 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.935+0000 7f733b86f700 1 --2- 172.21.15.19:0/3921759920 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7334100900 0x7f73341a01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:36.937 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.935+0000 7f733b86f700 1 --2- 172.21.15.19:0/3921759920 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f73340fff40 0x7f73341991b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:36.937 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.936+0000 7f733b86f700 1 -- 172.21.15.19:0/3921759920 >> 172.21.15.19:0/3921759920 conn(0x7f73340fb200 msgr2=0x7f73340fcd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:24:36.937 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.936+0000 7f733b86f700 1 -- 172.21.15.19:0/3921759920 shutdown_connections 2024-01-24T14:24:36.937 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:36.936+0000 7f733b86f700 1 -- 172.21.15.19:0/3921759920 wait complete. 2024-01-24T14:24:36.951 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T14:21:53.453429+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T14:24:36.951 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T14:24:37.133 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.132+0000 7f9d93311700 1 -- 172.21.15.19:0/3963242272 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9d8c0ffed0 msgr2=0x7f9d8c100330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:37.133 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.132+0000 7f9d93311700 1 --2- 172.21.15.19:0/3963242272 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9d8c0ffed0 0x7f9d8c100330 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c009990 tx=0x7f9d7c009ca0 comp rx=0 tx=0).stop 2024-01-24T14:24:37.133 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.132+0000 7f9d93311700 1 -- 172.21.15.19:0/3963242272 shutdown_connections 2024-01-24T14:24:37.134 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.132+0000 7f9d93311700 1 --2- 172.21.15.19:0/3963242272 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d8c100870 0x7f9d8c108130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:37.134 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.132+0000 7f9d93311700 1 --2- 172.21.15.19:0/3963242272 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d8c101230 0x7f9d8c101650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:37.134 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.132+0000 7f9d93311700 1 --2- 172.21.15.19:0/3963242272 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9d8c0ffed0 0x7f9d8c100330 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:37.134 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.132+0000 7f9d93311700 1 -- 172.21.15.19:0/3963242272 >> 172.21.15.19:0/3963242272 conn(0x7f9d8c0fa940 msgr2=0x7f9d8c0fcd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:24:37.134 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.132+0000 7f9d93311700 1 -- 172.21.15.19:0/3963242272 shutdown_connections 2024-01-24T14:24:37.134 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.132+0000 7f9d93311700 1 -- 172.21.15.19:0/3963242272 wait complete. 2024-01-24T14:24:37.134 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.132+0000 7f9d93311700 1 Processor -- start 2024-01-24T14:24:37.134 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.132+0000 7f9d93311700 1 -- start start 2024-01-24T14:24:37.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.133+0000 7f9d93311700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d8c0ffed0 0x7f9d8c103c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:37.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.133+0000 7f9d93311700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d8c100870 0x7f9d8c1022c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:37.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.133+0000 7f9d93311700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9d8c101230 0x7f9d8c102800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:37.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.133+0000 7f9d93311700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8c102fa0 con 0x7f9d8c101230 2024-01-24T14:24:37.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.133+0000 7f9d93311700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8c103110 con 0x7f9d8c0ffed0 2024-01-24T14:24:37.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.133+0000 7f9d93311700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8c1a0ab0 con 0x7f9d8c100870 2024-01-24T14:24:37.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.133+0000 7f9d918ae700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9d8c101230 0x7f9d8c102800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:37.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.133+0000 7f9d918ae700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9d8c101230 0x7f9d8c102800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:42752/0 (socket says 172.21.15.19:42752) 2024-01-24T14:24:37.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.133+0000 7f9d918ae700 1 -- 172.21.15.19:0/1250475994 learned_addr learned my addr 172.21.15.19:0/1250475994 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:24:37.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.133+0000 7f9d910ad700 1 --2- 172.21.15.19:0/1250475994 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d8c0ffed0 0x7f9d8c103c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:37.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.133+0000 7f9d908ac700 1 --2- 172.21.15.19:0/1250475994 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d8c100870 0x7f9d8c1022c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:37.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.133+0000 7f9d918ae700 1 -- 172.21.15.19:0/1250475994 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d8c100870 msgr2=0x7f9d8c1022c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:37.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.133+0000 7f9d918ae700 1 --2- 172.21.15.19:0/1250475994 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d8c100870 0x7f9d8c1022c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:37.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.133+0000 7f9d918ae700 1 -- 172.21.15.19:0/1250475994 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d8c0ffed0 msgr2=0x7f9d8c103c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:37.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.133+0000 7f9d918ae700 1 --2- 172.21.15.19:0/1250475994 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d8c0ffed0 0x7f9d8c103c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:37.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.133+0000 7f9d918ae700 1 -- 172.21.15.19:0/1250475994 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d7c0095f0 con 0x7f9d8c101230 2024-01-24T14:24:37.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.133+0000 7f9d918ae700 1 --2- 172.21.15.19:0/1250475994 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9d8c101230 0x7f9d8c102800 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9d8400ef30 tx=0x7f9d8400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:24:37.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.133+0000 7f9d827fc700 1 -- 172.21.15.19:0/1250475994 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d84018070 con 0x7f9d8c101230 2024-01-24T14:24:37.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.134+0000 7f9d827fc700 1 -- 172.21.15.19:0/1250475994 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9d8400f040 con 0x7f9d8c101230 2024-01-24T14:24:37.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.134+0000 7f9d93311700 1 -- 172.21.15.19:0/1250475994 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d8c1a0d30 con 0x7f9d8c101230 2024-01-24T14:24:37.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.134+0000 7f9d827fc700 1 -- 172.21.15.19:0/1250475994 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d8400cd00 con 0x7f9d8c101230 2024-01-24T14:24:37.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.134+0000 7f9d93311700 1 -- 172.21.15.19:0/1250475994 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d8c1a11e0 con 0x7f9d8c101230 2024-01-24T14:24:37.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.135+0000 7f9d93311700 1 -- 172.21.15.19:0/1250475994 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d8c10b4a0 con 0x7f9d8c101230 2024-01-24T14:24:37.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.136+0000 7f9d827fc700 1 -- 172.21.15.19:0/1250475994 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f9d84003a00 con 0x7f9d8c101230 2024-01-24T14:24:37.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.136+0000 7f9d827fc700 1 --2- 172.21.15.19:0/1250475994 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f9d780a05a0 0x7f9d780a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:37.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.136+0000 7f9d910ad700 1 --2- 172.21.15.19:0/1250475994 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f9d780a05a0 0x7f9d780a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:37.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.136+0000 7f9d827fc700 1 -- 172.21.15.19:0/1250475994 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f9d840cafc0 con 0x7f9d8c101230 2024-01-24T14:24:37.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.137+0000 7f9d910ad700 1 --2- 172.21.15.19:0/1250475994 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f9d780a05a0 0x7f9d780a2a60 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c068ed0 tx=0x7f9d88009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:24:37.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.141+0000 7f9d827fc700 1 -- 172.21.15.19:0/1250475994 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f9d840fe120 con 0x7f9d8c101230 2024-01-24T14:24:37.353 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.352+0000 7f9d93311700 1 -- 172.21.15.19:0/1250475994 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9d8c1a17f0 con 0x7f9d8c101230 2024-01-24T14:24:37.354 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.353+0000 7f9d827fc700 1 -- 172.21.15.19:0/1250475994 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v74) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9d84095090 con 0x7f9d8c101230 2024-01-24T14:24:37.354 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 1 2024-01-24T14:24:37.356 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.355+0000 7f9d93311700 1 -- 172.21.15.19:0/1250475994 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f9d780a05a0 msgr2=0x7f9d780a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:37.356 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.355+0000 7f9d93311700 1 --2- 172.21.15.19:0/1250475994 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f9d780a05a0 0x7f9d780a2a60 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c068ed0 tx=0x7f9d88009260 comp rx=0 tx=0).stop 2024-01-24T14:24:37.356 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.355+0000 7f9d93311700 1 -- 172.21.15.19:0/1250475994 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9d8c101230 msgr2=0x7f9d8c102800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:37.356 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.355+0000 7f9d93311700 1 --2- 172.21.15.19:0/1250475994 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9d8c101230 0x7f9d8c102800 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9d8400ef30 tx=0x7f9d8400c3a0 comp rx=0 tx=0).stop 2024-01-24T14:24:37.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.355+0000 7f9d93311700 1 -- 172.21.15.19:0/1250475994 shutdown_connections 2024-01-24T14:24:37.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.355+0000 7f9d93311700 1 --2- 172.21.15.19:0/1250475994 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f9d780a05a0 0x7f9d780a2a60 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:37.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.355+0000 7f9d93311700 1 --2- 172.21.15.19:0/1250475994 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9d8c101230 0x7f9d8c102800 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:37.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.355+0000 7f9d93311700 1 --2- 172.21.15.19:0/1250475994 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9d8c0ffed0 0x7f9d8c103c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:37.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.355+0000 7f9d93311700 1 --2- 172.21.15.19:0/1250475994 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d8c100870 0x7f9d8c1022c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:37.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.355+0000 7f9d93311700 1 -- 172.21.15.19:0/1250475994 >> 172.21.15.19:0/1250475994 conn(0x7f9d8c0fa940 msgr2=0x7f9d8c0fede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:24:37.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.355+0000 7f9d93311700 1 -- 172.21.15.19:0/1250475994 shutdown_connections 2024-01-24T14:24:37.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.355+0000 7f9d93311700 1 -- 172.21.15.19:0/1250475994 wait complete. 2024-01-24T14:24:37.370 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T14:24:37.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.507+0000 7f6e223bc700 1 -- 172.21.15.19:0/2139760976 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6e1c101820 msgr2=0x7f6e1c103c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:37.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.507+0000 7f6e223bc700 1 --2- 172.21.15.19:0/2139760976 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6e1c101820 0x7f6e1c103c10 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c009990 tx=0x7f6e0c009ca0 comp rx=0 tx=0).stop 2024-01-24T14:24:37.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.507+0000 7f6e223bc700 1 -- 172.21.15.19:0/2139760976 shutdown_connections 2024-01-24T14:24:37.509 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.507+0000 7f6e223bc700 1 --2- 172.21.15.19:0/2139760976 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6e1c101820 0x7f6e1c103c10 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:37.509 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.507+0000 7f6e223bc700 1 --2- 172.21.15.19:0/2139760976 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e1c106a80 0x7f6e1c108e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:37.509 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.507+0000 7f6e223bc700 1 --2- 172.21.15.19:0/2139760976 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6e1c104150 0x7f6e1c106540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:37.509 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.507+0000 7f6e223bc700 1 -- 172.21.15.19:0/2139760976 >> 172.21.15.19:0/2139760976 conn(0x7f6e1c0fb170 msgr2=0x7f6e1c0fd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:24:37.509 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.507+0000 7f6e223bc700 1 -- 172.21.15.19:0/2139760976 shutdown_connections 2024-01-24T14:24:37.509 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.507+0000 7f6e223bc700 1 -- 172.21.15.19:0/2139760976 wait complete. 2024-01-24T14:24:37.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.508+0000 7f6e223bc700 1 Processor -- start 2024-01-24T14:24:37.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.508+0000 7f6e223bc700 1 -- start start 2024-01-24T14:24:37.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.508+0000 7f6e223bc700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6e1c101820 0x7f6e1c196a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:37.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.508+0000 7f6e223bc700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e1c104150 0x7f6e1c196f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:37.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.508+0000 7f6e223bc700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6e1c106a80 0x7f6e1c19dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:37.512 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.508+0000 7f6e223bc700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1c19e580 con 0x7f6e1c101820 2024-01-24T14:24:37.512 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.508+0000 7f6e223bc700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1c19e6c0 con 0x7f6e1c104150 2024-01-24T14:24:37.512 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.508+0000 7f6e223bc700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1c19e850 con 0x7f6e1c106a80 2024-01-24T14:24:37.512 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.508+0000 7f6e1bfff700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6e1c101820 0x7f6e1c196a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:37.512 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.509+0000 7f6e1bfff700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6e1c101820 0x7f6e1c196a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:42764/0 (socket says 172.21.15.19:42764) 2024-01-24T14:24:37.512 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.509+0000 7f6e1bfff700 1 -- 172.21.15.19:0/3058065667 learned_addr learned my addr 172.21.15.19:0/3058065667 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:24:37.512 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.509+0000 7f6e20959700 1 --2- 172.21.15.19:0/3058065667 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6e1c106a80 0x7f6e1c19dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:37.512 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.509+0000 7f6e1b7fe700 1 --2- 172.21.15.19:0/3058065667 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e1c104150 0x7f6e1c196f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:37.512 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.509+0000 7f6e1bfff700 1 -- 172.21.15.19:0/3058065667 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6e1c106a80 msgr2=0x7f6e1c19dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:37.512 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.509+0000 7f6e1bfff700 1 --2- 172.21.15.19:0/3058065667 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6e1c106a80 0x7f6e1c19dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:37.512 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.509+0000 7f6e1bfff700 1 -- 172.21.15.19:0/3058065667 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e1c104150 msgr2=0x7f6e1c196f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:37.512 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.509+0000 7f6e1bfff700 1 --2- 172.21.15.19:0/3058065667 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e1c104150 0x7f6e1c196f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:37.512 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.509+0000 7f6e1bfff700 1 -- 172.21.15.19:0/3058065667 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e0c0095f0 con 0x7f6e1c101820 2024-01-24T14:24:37.513 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.509+0000 7f6e20959700 1 --2- 172.21.15.19:0/3058065667 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6e1c106a80 0x7f6e1c19dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T14:24:37.513 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.509+0000 7f6e1b7fe700 1 --2- 172.21.15.19:0/3058065667 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e1c104150 0x7f6e1c196f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T14:24:37.513 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.509+0000 7f6e1bfff700 1 --2- 172.21.15.19:0/3058065667 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6e1c101820 0x7f6e1c196a00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c004730 tx=0x7f6e0c004760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:24:37.513 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.509+0000 7f6e197fa700 1 -- 172.21.15.19:0/3058065667 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e0c01d070 con 0x7f6e1c101820 2024-01-24T14:24:37.513 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.509+0000 7f6e223bc700 1 -- 172.21.15.19:0/3058065667 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e1c19ead0 con 0x7f6e1c101820 2024-01-24T14:24:37.513 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.509+0000 7f6e197fa700 1 -- 172.21.15.19:0/3058065667 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6e0c004c90 con 0x7f6e1c101820 2024-01-24T14:24:37.513 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.509+0000 7f6e197fa700 1 -- 172.21.15.19:0/3058065667 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e0c005570 con 0x7f6e1c101820 2024-01-24T14:24:37.513 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.509+0000 7f6e223bc700 1 -- 172.21.15.19:0/3058065667 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e1c19eea0 con 0x7f6e1c101820 2024-01-24T14:24:37.513 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.510+0000 7f6e223bc700 1 -- 172.21.15.19:0/3058065667 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e1c0fccf0 con 0x7f6e1c101820 2024-01-24T14:24:37.514 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.511+0000 7f6e197fa700 1 -- 172.21.15.19:0/3058065667 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f6e0c00fc70 con 0x7f6e1c101820 2024-01-24T14:24:37.514 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.512+0000 7f6e197fa700 1 --2- 172.21.15.19:0/3058065667 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f6e080a0570 0x7f6e080a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:37.514 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.512+0000 7f6e1b7fe700 1 --2- 172.21.15.19:0/3058065667 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f6e080a0570 0x7f6e080a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:37.514 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.512+0000 7f6e197fa700 1 -- 172.21.15.19:0/3058065667 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f6e0c0cc8c0 con 0x7f6e1c101820 2024-01-24T14:24:37.514 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.513+0000 7f6e1b7fe700 1 --2- 172.21.15.19:0/3058065667 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f6e080a0570 0x7f6e080a2a30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6e04009cb0 tx=0x7f6e04009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:24:37.517 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.516+0000 7f6e197fa700 1 -- 172.21.15.19:0/3058065667 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f6e0c05b120 con 0x7f6e1c101820 2024-01-24T14:24:37.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:37 smithi194 ceph-mon[45450]: from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi019=a;smithi019=d;smithi019=g;smithi019=j;smithi123=b;smithi123=e;smithi123=h;smithi123=k;smithi194=c;smithi194=f;smithi194=i;smithi194=l", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:24:37.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:37 smithi194 ceph-mon[45450]: Saving service mds.all spec with placement smithi019=a;smithi019=d;smithi019=g;smithi019=j;smithi123=b;smithi123=e;smithi123=h;smithi123=k;smithi194=c;smithi194=f;smithi194=i;smithi194=l;count:12 2024-01-24T14:24:37.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:37 smithi194 ceph-mon[45450]: Deploying daemon mds.l on smithi194 2024-01-24T14:24:37.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:37 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/3921759920' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T14:24:37.605 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:37 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/1250475994' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-01-24T14:24:37.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:37 smithi019 ceph-mon[40663]: from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi019=a;smithi019=d;smithi019=g;smithi019=j;smithi123=b;smithi123=e;smithi123=h;smithi123=k;smithi194=c;smithi194=f;smithi194=i;smithi194=l", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:24:37.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:37 smithi019 ceph-mon[40663]: Saving service mds.all spec with placement smithi019=a;smithi019=d;smithi019=g;smithi019=j;smithi123=b;smithi123=e;smithi123=h;smithi123=k;smithi194=c;smithi194=f;smithi194=i;smithi194=l;count:12 2024-01-24T14:24:37.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:37 smithi019 ceph-mon[40663]: Deploying daemon mds.l on smithi194 2024-01-24T14:24:37.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:37 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/3921759920' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T14:24:37.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:37 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/1250475994' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-01-24T14:24:37.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.775+0000 7f6e223bc700 1 -- 172.21.15.19:0/3058065667 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e1c02ce70 con 0x7f6e1c101820 2024-01-24T14:24:37.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.776+0000 7f6e197fa700 1 -- 172.21.15.19:0/3058065667 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+501 (secure 0 0 0) 0x7f6e0c096990 con 0x7f6e1c101820 2024-01-24T14:24:37.777 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:24:37.777 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-01-24T14:24:37.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.778+0000 7f6e223bc700 1 -- 172.21.15.19:0/3058065667 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f6e080a0570 msgr2=0x7f6e080a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:37.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.778+0000 7f6e223bc700 1 --2- 172.21.15.19:0/3058065667 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f6e080a0570 0x7f6e080a2a30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6e04009cb0 tx=0x7f6e04009260 comp rx=0 tx=0).stop 2024-01-24T14:24:37.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.778+0000 7f6e223bc700 1 -- 172.21.15.19:0/3058065667 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6e1c101820 msgr2=0x7f6e1c196a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:37.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.778+0000 7f6e223bc700 1 --2- 172.21.15.19:0/3058065667 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6e1c101820 0x7f6e1c196a00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c004730 tx=0x7f6e0c004760 comp rx=0 tx=0).stop 2024-01-24T14:24:37.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.778+0000 7f6e223bc700 1 -- 172.21.15.19:0/3058065667 shutdown_connections 2024-01-24T14:24:37.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.778+0000 7f6e223bc700 1 --2- 172.21.15.19:0/3058065667 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f6e080a0570 0x7f6e080a2a30 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:37.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.778+0000 7f6e223bc700 1 --2- 172.21.15.19:0/3058065667 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6e1c101820 0x7f6e1c196a00 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:37.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.778+0000 7f6e223bc700 1 --2- 172.21.15.19:0/3058065667 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6e1c106a80 0x7f6e1c19dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:37.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.778+0000 7f6e223bc700 1 --2- 172.21.15.19:0/3058065667 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6e1c104150 0x7f6e1c196f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:37.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.778+0000 7f6e223bc700 1 -- 172.21.15.19:0/3058065667 >> 172.21.15.19:0/3058065667 conn(0x7f6e1c0fb170 msgr2=0x7f6e1c102490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:24:37.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.779+0000 7f6e223bc700 1 -- 172.21.15.19:0/3058065667 shutdown_connections 2024-01-24T14:24:37.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:37.779+0000 7f6e223bc700 1 -- 172.21.15.19:0/3058065667 wait complete. 2024-01-24T14:24:37.781 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 1 2024-01-24T14:24:37.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:37 smithi123 ceph-mon[44966]: from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi019=a;smithi019=d;smithi019=g;smithi019=j;smithi123=b;smithi123=e;smithi123=h;smithi123=k;smithi194=c;smithi194=f;smithi194=i;smithi194=l", "target": ["mon-mgr", ""]}]: dispatch 2024-01-24T14:24:37.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:37 smithi123 ceph-mon[44966]: Saving service mds.all spec with placement smithi019=a;smithi019=d;smithi019=g;smithi019=j;smithi123=b;smithi123=e;smithi123=h;smithi123=k;smithi194=c;smithi194=f;smithi194=i;smithi194=l;count:12 2024-01-24T14:24:37.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:37 smithi123 ceph-mon[44966]: Deploying daemon mds.l on smithi194 2024-01-24T14:24:37.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:37 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/3921759920' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T14:24:37.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:37 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/1250475994' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-01-24T14:24:38.344 INFO:journalctl@ceph.mds.l.smithi194.stdout:Jan 24 14:24:37 smithi194 bash[82049]: 5c2d0c26405039d9570b073afd1d3c2bbaf513cfb7cffdef5771fb6e1cd98698 2024-01-24T14:24:38.344 INFO:journalctl@ceph.mds.l.smithi194.stdout:Jan 24 14:24:37 smithi194 systemd[1]: Started Ceph mds.l for 654b0c8e-bac3-11ee-95b1-87774f69a715. 2024-01-24T14:24:38.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:38 smithi019 ceph-mon[40663]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:24:38.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:38 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/3058065667' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T14:24:38.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:38 smithi019 ceph-mon[40663]: mds.? [v2:172.21.15.194:6832/2647625404,v1:172.21.15.194:6833/2647625404] up:boot 2024-01-24T14:24:38.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:38 smithi019 ceph-mon[40663]: fsmap 1 up:standby 2024-01-24T14:24:38.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:38 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-01-24T14:24:38.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:38 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:38.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:38 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:38.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:38 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:38.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:38 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T14:24:38.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:38 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T14:24:38.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:38 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:38.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:38 smithi123 ceph-mon[44966]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:24:38.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:38 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/3058065667' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T14:24:38.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:38 smithi123 ceph-mon[44966]: mds.? [v2:172.21.15.194:6832/2647625404,v1:172.21.15.194:6833/2647625404] up:boot 2024-01-24T14:24:38.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:38 smithi123 ceph-mon[44966]: fsmap 1 up:standby 2024-01-24T14:24:38.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:38 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-01-24T14:24:38.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:38 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:38.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:38 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:38.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:38 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:38.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:38 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T14:24:38.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:38 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T14:24:38.771 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:38 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:38.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:38 smithi194 ceph-mon[45450]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:24:38.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:38 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/3058065667' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T14:24:38.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:38 smithi194 ceph-mon[45450]: mds.? [v2:172.21.15.194:6832/2647625404,v1:172.21.15.194:6833/2647625404] up:boot 2024-01-24T14:24:38.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:38 smithi194 ceph-mon[45450]: fsmap 1 up:standby 2024-01-24T14:24:38.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:38 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-01-24T14:24:38.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:38 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:38.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:38 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:38.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:38 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:38.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:38 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T14:24:38.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:38 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T14:24:38.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:38 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:39.445 INFO:journalctl@ceph.mds.k.smithi123.stdout:Jan 24 14:24:39 smithi123 systemd[1]: Starting Ceph mds.k for 654b0c8e-bac3-11ee-95b1-87774f69a715... 2024-01-24T14:24:39.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:39 smithi019 ceph-mon[40663]: Deploying daemon mds.k on smithi123 2024-01-24T14:24:39.793 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T14:24:39.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:39 smithi123 ceph-mon[44966]: Deploying daemon mds.k on smithi123 2024-01-24T14:24:39.799 INFO:journalctl@ceph.mds.k.smithi123.stdout:Jan 24 14:24:39 smithi123 ceph-mds[80347]: -- 172.21.15.123:0/1608438558 <== mon.2 v2:172.21.15.123:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5639ed7b91e0 con 0x5639ed78ec00 2024-01-24T14:24:39.799 INFO:journalctl@ceph.mds.k.smithi123.stdout:Jan 24 14:24:39 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-k[80324]: starting mds.k at 2024-01-24T14:24:39.799 INFO:journalctl@ceph.mds.k.smithi123.stdout:Jan 24 14:24:39 smithi123 bash[80309]: 9f429433039a38517fc1fd475beaeb81e03dfe14e20f48489b9507e3bc0acbf8 2024-01-24T14:24:39.799 INFO:journalctl@ceph.mds.k.smithi123.stdout:Jan 24 14:24:39 smithi123 systemd[1]: Started Ceph mds.k for 654b0c8e-bac3-11ee-95b1-87774f69a715. 2024-01-24T14:24:39.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:39 smithi194 ceph-mon[45450]: Deploying daemon mds.k on smithi123 2024-01-24T14:24:39.947 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:39.944+0000 7fc57aeda700 1 -- 172.21.15.19:0/2580917058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc574101800 msgr2=0x7fc574103bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:39.947 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:39.944+0000 7fc57aeda700 1 --2- 172.21.15.19:0/2580917058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc574101800 0x7fc574103bf0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc568009990 tx=0x7fc568009ca0 comp rx=0 tx=0).stop 2024-01-24T14:24:39.948 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:39.944+0000 7fc57aeda700 1 -- 172.21.15.19:0/2580917058 shutdown_connections 2024-01-24T14:24:39.948 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:39.944+0000 7fc57aeda700 1 --2- 172.21.15.19:0/2580917058 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc574106a60 0x7fc574108e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:39.948 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:39.944+0000 7fc57aeda700 1 --2- 172.21.15.19:0/2580917058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc574101800 0x7fc574103bf0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:39.948 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:39.944+0000 7fc57aeda700 1 --2- 172.21.15.19:0/2580917058 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc574104130 0x7fc574106520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:39.948 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:39.944+0000 7fc57aeda700 1 -- 172.21.15.19:0/2580917058 >> 172.21.15.19:0/2580917058 conn(0x7fc5740fb170 msgr2=0x7fc5740fd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:24:39.948 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:39.944+0000 7fc57aeda700 1 -- 172.21.15.19:0/2580917058 shutdown_connections 2024-01-24T14:24:39.948 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:39.944+0000 7fc57aeda700 1 -- 172.21.15.19:0/2580917058 wait complete. 2024-01-24T14:24:39.948 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:39.945+0000 7fc57aeda700 1 Processor -- start 2024-01-24T14:24:39.948 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:39.945+0000 7fc57aeda700 1 -- start start 2024-01-24T14:24:39.948 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:39.945+0000 7fc57aeda700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc574101800 0x7fc574198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:39.948 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:39.945+0000 7fc57aeda700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc574104130 0x7fc5741990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:39.948 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:39.945+0000 7fc57aeda700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc574106a60 0x7fc5741a0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:39.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:39.945+0000 7fc57aeda700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5741a0730 con 0x7fc574104130 2024-01-24T14:24:39.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:39.945+0000 7fc57aeda700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5741a0870 con 0x7fc574101800 2024-01-24T14:24:39.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:39.945+0000 7fc57aeda700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5741a0a00 con 0x7fc574106a60 2024-01-24T14:24:39.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:39.945+0000 7fc578c76700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc574101800 0x7fc574198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:39.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:39.945+0000 7fc578c76700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc574101800 0x7fc574198bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.19:58248/0 (socket says 172.21.15.19:58248) 2024-01-24T14:24:39.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:39.945+0000 7fc578c76700 1 -- 172.21.15.19:0/4112579032 learned_addr learned my addr 172.21.15.19:0/4112579032 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:24:39.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:39.945+0000 7fc573fff700 1 --2- 172.21.15.19:0/4112579032 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc574104130 0x7fc5741990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:39.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:39.945+0000 7fc579477700 1 --2- 172.21.15.19:0/4112579032 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc574106a60 0x7fc5741a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:39.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:39.945+0000 7fc578c76700 1 -- 172.21.15.19:0/4112579032 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc574106a60 msgr2=0x7fc5741a0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:39.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:39.945+0000 7fc578c76700 1 --2- 172.21.15.19:0/4112579032 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc574106a60 0x7fc5741a0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:39.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:39.945+0000 7fc578c76700 1 -- 172.21.15.19:0/4112579032 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc574104130 msgr2=0x7fc5741990f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:39.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:39.945+0000 7fc578c76700 1 --2- 172.21.15.19:0/4112579032 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc574104130 0x7fc5741990f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:39.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:39.945+0000 7fc578c76700 1 -- 172.21.15.19:0/4112579032 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5680095f0 con 0x7fc574101800 2024-01-24T14:24:39.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:39.945+0000 7fc579477700 1 --2- 172.21.15.19:0/4112579032 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc574106a60 0x7fc5741a0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T14:24:39.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:39.946+0000 7fc578c76700 1 --2- 172.21.15.19:0/4112579032 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc574101800 0x7fc574198bb0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc568004730 tx=0x7fc568004760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:24:39.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:39.946+0000 7fc571ffb700 1 -- 172.21.15.19:0/4112579032 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc56801d070 con 0x7fc574101800 2024-01-24T14:24:39.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:39.946+0000 7fc571ffb700 1 -- 172.21.15.19:0/4112579032 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc568004c90 con 0x7fc574101800 2024-01-24T14:24:39.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:39.946+0000 7fc57aeda700 1 -- 172.21.15.19:0/4112579032 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5741a0c80 con 0x7fc574101800 2024-01-24T14:24:39.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:39.946+0000 7fc571ffb700 1 -- 172.21.15.19:0/4112579032 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc568005570 con 0x7fc574101800 2024-01-24T14:24:39.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:39.946+0000 7fc57aeda700 1 -- 172.21.15.19:0/4112579032 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5741a10a0 con 0x7fc574101800 2024-01-24T14:24:39.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:39.947+0000 7fc57aeda700 1 -- 172.21.15.19:0/4112579032 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5741085e0 con 0x7fc574101800 2024-01-24T14:24:39.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:39.948+0000 7fc571ffb700 1 -- 172.21.15.19:0/4112579032 <== mon.2 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7fc5680047d0 con 0x7fc574101800 2024-01-24T14:24:39.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:39.949+0000 7fc571ffb700 1 --2- 172.21.15.19:0/4112579032 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fc5640a05f0 0x7fc5640a2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:39.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:39.949+0000 7fc571ffb700 1 -- 172.21.15.19:0/4112579032 <== mon.2 v2:172.21.15.123:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc5680cc8b0 con 0x7fc574101800 2024-01-24T14:24:39.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:39.949+0000 7fc573fff700 1 --2- 172.21.15.19:0/4112579032 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fc5640a05f0 0x7fc5640a2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:39.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:39.949+0000 7fc573fff700 1 --2- 172.21.15.19:0/4112579032 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fc5640a05f0 0x7fc5640a2ab0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc560007c60 tx=0x7fc5600073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:24:39.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:39.953+0000 7fc571ffb700 1 -- 172.21.15.19:0/4112579032 <== mon.2 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fc56805b120 con 0x7fc574101800 2024-01-24T14:24:40.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:40.236+0000 7fc57aeda700 1 -- 172.21.15.19:0/4112579032 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc57402d090 con 0x7fc574101800 2024-01-24T14:24:40.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:40.236+0000 7fc571ffb700 1 -- 172.21.15.19:0/4112579032 <== mon.2 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1260 (secure 0 0 0) 0x7fc5680968e0 con 0x7fc574101800 2024-01-24T14:24:40.237 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:24:40.237 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2647625404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2647625404},{"type":"v1","addr":"172.21.15.194:6833","nonce":2647625404}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2024-01-24T14:24:40.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:40.239+0000 7fc57aeda700 1 -- 172.21.15.19:0/4112579032 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fc5640a05f0 msgr2=0x7fc5640a2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:40.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:40.239+0000 7fc57aeda700 1 --2- 172.21.15.19:0/4112579032 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fc5640a05f0 0x7fc5640a2ab0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc560007c60 tx=0x7fc5600073d0 comp rx=0 tx=0).stop 2024-01-24T14:24:40.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:40.239+0000 7fc57aeda700 1 -- 172.21.15.19:0/4112579032 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc574101800 msgr2=0x7fc574198bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:40.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:40.239+0000 7fc57aeda700 1 --2- 172.21.15.19:0/4112579032 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc574101800 0x7fc574198bb0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc568004730 tx=0x7fc568004760 comp rx=0 tx=0).stop 2024-01-24T14:24:40.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:40.239+0000 7fc57aeda700 1 -- 172.21.15.19:0/4112579032 shutdown_connections 2024-01-24T14:24:40.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:40.239+0000 7fc57aeda700 1 --2- 172.21.15.19:0/4112579032 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc574106a60 0x7fc5741a0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:40.241 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:40.239+0000 7fc57aeda700 1 --2- 172.21.15.19:0/4112579032 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc574101800 0x7fc574198bb0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:40.241 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:40.239+0000 7fc57aeda700 1 --2- 172.21.15.19:0/4112579032 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fc5640a05f0 0x7fc5640a2ab0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:40.241 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:40.239+0000 7fc57aeda700 1 --2- 172.21.15.19:0/4112579032 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc574104130 0x7fc5741990f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:40.241 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:40.239+0000 7fc57aeda700 1 -- 172.21.15.19:0/4112579032 >> 172.21.15.19:0/4112579032 conn(0x7fc5740fb170 msgr2=0x7fc5740ffe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:24:40.241 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:40.239+0000 7fc57aeda700 1 -- 172.21.15.19:0/4112579032 shutdown_connections 2024-01-24T14:24:40.241 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:40.239+0000 7fc57aeda700 1 -- 172.21.15.19:0/4112579032 wait complete. 2024-01-24T14:24:40.241 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 2 2024-01-24T14:24:40.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:40 smithi019 ceph-mon[40663]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:24:40.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:40 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:40.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:40 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:40.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:40 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:40.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:40 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T14:24:40.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:40 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T14:24:40.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:40 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:40.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:40 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/4112579032' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T14:24:40.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:40 smithi123 ceph-mon[44966]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:24:40.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:40 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:40.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:40 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:40.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:40 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:40.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:40 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T14:24:40.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:40 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T14:24:40.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:40 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:40.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:40 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/4112579032' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T14:24:40.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:40 smithi194 ceph-mon[45450]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:24:40.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:40 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:40.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:40 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:40.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:40 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:40.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:40 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T14:24:40.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:40 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T14:24:40.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:40 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:40.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:40 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/4112579032' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T14:24:41.457 INFO:journalctl@ceph.mds.j.smithi019.stdout:Jan 24 14:24:41 smithi019 systemd[1]: Starting Ceph mds.j for 654b0c8e-bac3-11ee-95b1-87774f69a715... 2024-01-24T14:24:41.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:41 smithi019 ceph-mon[40663]: Deploying daemon mds.j on smithi019 2024-01-24T14:24:41.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:41 smithi019 ceph-mon[40663]: mds.? [v2:172.21.15.123:6832/3396650565,v1:172.21.15.123:6833/3396650565] up:boot 2024-01-24T14:24:41.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:41 smithi019 ceph-mon[40663]: fsmap 2 up:standby 2024-01-24T14:24:41.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:41 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-01-24T14:24:41.737 INFO:journalctl@ceph.mds.j.smithi019.stdout:Jan 24 14:24:41 smithi019 ceph-mds[92200]: -- 172.21.15.19:0/4127283014 <== mon.0 v2:172.21.15.19:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e6b0e6d1e0 con 0x55e6b0e42c00 2024-01-24T14:24:41.737 INFO:journalctl@ceph.mds.j.smithi019.stdout:Jan 24 14:24:41 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-j[92177]: starting mds.j at 2024-01-24T14:24:41.737 INFO:journalctl@ceph.mds.j.smithi019.stdout:Jan 24 14:24:41 smithi019 bash[92162]: 7b4b8cf4c49d9678d9134421754f76abbd41d5c55a891409a8eb85ec4d1cc76b 2024-01-24T14:24:41.737 INFO:journalctl@ceph.mds.j.smithi019.stdout:Jan 24 14:24:41 smithi019 systemd[1]: Started Ceph mds.j for 654b0c8e-bac3-11ee-95b1-87774f69a715. 2024-01-24T14:24:41.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:41 smithi123 ceph-mon[44966]: Deploying daemon mds.j on smithi019 2024-01-24T14:24:41.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:41 smithi123 ceph-mon[44966]: mds.? [v2:172.21.15.123:6832/3396650565,v1:172.21.15.123:6833/3396650565] up:boot 2024-01-24T14:24:41.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:41 smithi123 ceph-mon[44966]: fsmap 2 up:standby 2024-01-24T14:24:41.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:41 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-01-24T14:24:41.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:41 smithi194 ceph-mon[45450]: Deploying daemon mds.j on smithi019 2024-01-24T14:24:41.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:41 smithi194 ceph-mon[45450]: mds.? [v2:172.21.15.123:6832/3396650565,v1:172.21.15.123:6833/3396650565] up:boot 2024-01-24T14:24:41.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:41 smithi194 ceph-mon[45450]: fsmap 2 up:standby 2024-01-24T14:24:41.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:41 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-01-24T14:24:42.255 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T14:24:42.408 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.405+0000 7fe537ada700 1 -- 172.21.15.19:0/656841307 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe530106a00 msgr2=0x7fe530108e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:42.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.405+0000 7fe537ada700 1 --2- 172.21.15.19:0/656841307 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe530106a00 0x7fe530108e70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe52800b0a0 tx=0x7fe52800b3b0 comp rx=0 tx=0).stop 2024-01-24T14:24:42.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.405+0000 7fe537ada700 1 -- 172.21.15.19:0/656841307 shutdown_connections 2024-01-24T14:24:42.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.405+0000 7fe537ada700 1 --2- 172.21.15.19:0/656841307 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe530106a00 0x7fe530108e70 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:42.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.405+0000 7fe537ada700 1 --2- 172.21.15.19:0/656841307 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5301017a0 0x7fe530103b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:42.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.405+0000 7fe537ada700 1 --2- 172.21.15.19:0/656841307 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe5301040d0 0x7fe5301064c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:42.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.405+0000 7fe537ada700 1 -- 172.21.15.19:0/656841307 >> 172.21.15.19:0/656841307 conn(0x7fe5300fb170 msgr2=0x7fe5300fd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:24:42.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.405+0000 7fe537ada700 1 -- 172.21.15.19:0/656841307 shutdown_connections 2024-01-24T14:24:42.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.405+0000 7fe537ada700 1 -- 172.21.15.19:0/656841307 wait complete. 2024-01-24T14:24:42.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.406+0000 7fe537ada700 1 Processor -- start 2024-01-24T14:24:42.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.406+0000 7fe537ada700 1 -- start start 2024-01-24T14:24:42.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.406+0000 7fe537ada700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5301017a0 0x7fe530198d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:42.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.406+0000 7fe537ada700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe5301040d0 0x7fe530199250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:42.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.406+0000 7fe537ada700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe530106a00 0x7fe5301a0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:42.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.406+0000 7fe537ada700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5301a0850 con 0x7fe530106a00 2024-01-24T14:24:42.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.406+0000 7fe537ada700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5301a09c0 con 0x7fe5301017a0 2024-01-24T14:24:42.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.406+0000 7fe537ada700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5301a0b80 con 0x7fe5301040d0 2024-01-24T14:24:42.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.406+0000 7fe536077700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe530106a00 0x7fe5301a0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:42.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.406+0000 7fe536077700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe530106a00 0x7fe5301a0280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:51694/0 (socket says 172.21.15.19:51694) 2024-01-24T14:24:42.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.406+0000 7fe536077700 1 -- 172.21.15.19:0/3974323326 learned_addr learned my addr 172.21.15.19:0/3974323326 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:24:42.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.406+0000 7fe535876700 1 --2- 172.21.15.19:0/3974323326 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5301017a0 0x7fe530198d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:42.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.406+0000 7fe535075700 1 --2- 172.21.15.19:0/3974323326 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe5301040d0 0x7fe530199250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:42.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.406+0000 7fe536077700 1 -- 172.21.15.19:0/3974323326 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe5301040d0 msgr2=0x7fe530199250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:42.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.406+0000 7fe536077700 1 --2- 172.21.15.19:0/3974323326 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe5301040d0 0x7fe530199250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:42.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.406+0000 7fe536077700 1 -- 172.21.15.19:0/3974323326 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5301017a0 msgr2=0x7fe530198d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:42.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.406+0000 7fe536077700 1 --2- 172.21.15.19:0/3974323326 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5301017a0 0x7fe530198d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:42.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.406+0000 7fe536077700 1 -- 172.21.15.19:0/3974323326 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe528009d00 con 0x7fe530106a00 2024-01-24T14:24:42.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.406+0000 7fe536077700 1 --2- 172.21.15.19:0/3974323326 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe530106a00 0x7fe5301a0280 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe528009fd0 tx=0x7fe52800bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:24:42.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.407+0000 7fe526ffd700 1 -- 172.21.15.19:0/3974323326 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe52800e040 con 0x7fe530106a00 2024-01-24T14:24:42.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.407+0000 7fe526ffd700 1 -- 172.21.15.19:0/3974323326 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe528003e10 con 0x7fe530106a00 2024-01-24T14:24:42.412 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.407+0000 7fe537ada700 1 -- 172.21.15.19:0/3974323326 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5301a0e00 con 0x7fe530106a00 2024-01-24T14:24:42.413 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.407+0000 7fe526ffd700 1 -- 172.21.15.19:0/3974323326 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe52801b750 con 0x7fe530106a00 2024-01-24T14:24:42.413 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.407+0000 7fe537ada700 1 -- 172.21.15.19:0/3974323326 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5301a11d0 con 0x7fe530106a00 2024-01-24T14:24:42.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.408+0000 7fe537ada700 1 -- 172.21.15.19:0/3974323326 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe530108580 con 0x7fe530106a00 2024-01-24T14:24:42.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.409+0000 7fe526ffd700 1 -- 172.21.15.19:0/3974323326 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7fe52801da30 con 0x7fe530106a00 2024-01-24T14:24:42.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.410+0000 7fe526ffd700 1 --2- 172.21.15.19:0/3974323326 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fe51c0a0570 0x7fe51c0a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:42.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.410+0000 7fe535876700 1 --2- 172.21.15.19:0/3974323326 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fe51c0a0570 0x7fe51c0a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:42.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.410+0000 7fe526ffd700 1 -- 172.21.15.19:0/3974323326 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe5280cce50 con 0x7fe530106a00 2024-01-24T14:24:42.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.410+0000 7fe535876700 1 --2- 172.21.15.19:0/3974323326 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fe51c0a0570 0x7fe51c0a2a30 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe53019a1e0 tx=0x7fe52c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:24:42.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.413+0000 7fe526ffd700 1 -- 172.21.15.19:0/3974323326 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fe528096e80 con 0x7fe530106a00 2024-01-24T14:24:42.572 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:42 smithi019 ceph-mon[40663]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:24:42.572 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:42 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:42.573 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:42 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:42.573 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:42 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:42.573 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:42 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T14:24:42.573 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:42 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T14:24:42.573 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:42 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:42.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.688+0000 7fe537ada700 1 -- 172.21.15.19:0/3974323326 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe53002cc70 con 0x7fe530106a00 2024-01-24T14:24:42.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.688+0000 7fe526ffd700 1 -- 172.21.15.19:0/3974323326 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2777 (secure 0 0 0) 0x7fe52809a4a0 con 0x7fe530106a00 2024-01-24T14:24:42.690 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:24:42.690 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6835/1060583512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6834","nonce":1060583512},{"type":"v1","addr":"172.21.15.19:6835","nonce":1060583512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2647625404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2647625404},{"type":"v1","addr":"172.21.15.194:6833","nonce":2647625404}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/3396650565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3396650565},{"type":"v1","addr":"172.21.15.123:6833","nonce":3396650565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2024-01-24T14:24:42.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.691+0000 7fe537ada700 1 -- 172.21.15.19:0/3974323326 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fe51c0a0570 msgr2=0x7fe51c0a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:42.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.691+0000 7fe537ada700 1 --2- 172.21.15.19:0/3974323326 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fe51c0a0570 0x7fe51c0a2a30 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe53019a1e0 tx=0x7fe52c009260 comp rx=0 tx=0).stop 2024-01-24T14:24:42.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.691+0000 7fe537ada700 1 -- 172.21.15.19:0/3974323326 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe530106a00 msgr2=0x7fe5301a0280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:42.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.691+0000 7fe537ada700 1 --2- 172.21.15.19:0/3974323326 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe530106a00 0x7fe5301a0280 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe528009fd0 tx=0x7fe52800bce0 comp rx=0 tx=0).stop 2024-01-24T14:24:42.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.691+0000 7fe537ada700 1 -- 172.21.15.19:0/3974323326 shutdown_connections 2024-01-24T14:24:42.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.691+0000 7fe537ada700 1 --2- 172.21.15.19:0/3974323326 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fe51c0a0570 0x7fe51c0a2a30 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:42.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.691+0000 7fe537ada700 1 --2- 172.21.15.19:0/3974323326 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe530106a00 0x7fe5301a0280 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:42.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.691+0000 7fe537ada700 1 --2- 172.21.15.19:0/3974323326 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5301017a0 0x7fe530198d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:42.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.691+0000 7fe537ada700 1 --2- 172.21.15.19:0/3974323326 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe5301040d0 0x7fe530199250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:42.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.691+0000 7fe537ada700 1 -- 172.21.15.19:0/3974323326 >> 172.21.15.19:0/3974323326 conn(0x7fe5300fb170 msgr2=0x7fe5300ffe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:24:42.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.692+0000 7fe537ada700 1 -- 172.21.15.19:0/3974323326 shutdown_connections 2024-01-24T14:24:42.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:42.692+0000 7fe537ada700 1 -- 172.21.15.19:0/3974323326 wait complete. 2024-01-24T14:24:42.695 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 4 2024-01-24T14:24:42.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:42 smithi123 ceph-mon[44966]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:24:42.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:42 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:42.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:42 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:42.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:42 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:42.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:42 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T14:24:42.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:42 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T14:24:42.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:42 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:42.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:42 smithi194 ceph-mon[45450]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:24:42.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:42 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:42.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:42 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:42.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:42 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:42.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:42 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T14:24:42.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:42 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T14:24:42.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:42 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:43.131 INFO:journalctl@ceph.mds.d.smithi019.stdout:Jan 24 14:24:42 smithi019 systemd[1]: Starting Ceph mds.d for 654b0c8e-bac3-11ee-95b1-87774f69a715... 2024-01-24T14:24:43.736 INFO:journalctl@ceph.mds.d.smithi019.stdout:Jan 24 14:24:43 smithi019 ceph-mds[92588]: -- 172.21.15.19:0/4226213648 <== mon.0 v2:172.21.15.19:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55fa505dd1e0 con 0x55fa505b3000 2024-01-24T14:24:43.736 INFO:journalctl@ceph.mds.d.smithi019.stdout:Jan 24 14:24:43 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-d[92564]: starting mds.d at 2024-01-24T14:24:43.736 INFO:journalctl@ceph.mds.d.smithi019.stdout:Jan 24 14:24:43 smithi019 bash[92550]: 6127fcdb3babf6cc3203ca7af80c6f7205ff9ae81b84e18c6e4fe82ad8f9e7a9 2024-01-24T14:24:43.736 INFO:journalctl@ceph.mds.d.smithi019.stdout:Jan 24 14:24:43 smithi019 systemd[1]: Started Ceph mds.d for 654b0c8e-bac3-11ee-95b1-87774f69a715. 2024-01-24T14:24:43.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:43 smithi019 ceph-mon[40663]: Deploying daemon mds.d on smithi019 2024-01-24T14:24:43.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:43 smithi019 ceph-mon[40663]: mds.? [v2:172.21.15.19:6834/1060583512,v1:172.21.15.19:6835/1060583512] up:boot 2024-01-24T14:24:43.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:43 smithi019 ceph-mon[40663]: fsmap 3 up:standby 2024-01-24T14:24:43.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:43 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-01-24T14:24:43.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:43 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/3974323326' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T14:24:43.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:43 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:43.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:43 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:43.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:43 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:43.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:43 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T14:24:43.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:43 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T14:24:43.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:43 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:43.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:43 smithi123 ceph-mon[44966]: Deploying daemon mds.d on smithi019 2024-01-24T14:24:43.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:43 smithi123 ceph-mon[44966]: mds.? [v2:172.21.15.19:6834/1060583512,v1:172.21.15.19:6835/1060583512] up:boot 2024-01-24T14:24:43.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:43 smithi123 ceph-mon[44966]: fsmap 3 up:standby 2024-01-24T14:24:43.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:43 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-01-24T14:24:43.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:43 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/3974323326' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T14:24:43.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:43 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:43.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:43 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:43.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:43 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:43.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:43 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T14:24:43.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:43 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T14:24:43.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:43 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:43.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:43 smithi194 ceph-mon[45450]: Deploying daemon mds.d on smithi019 2024-01-24T14:24:43.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:43 smithi194 ceph-mon[45450]: mds.? [v2:172.21.15.19:6834/1060583512,v1:172.21.15.19:6835/1060583512] up:boot 2024-01-24T14:24:43.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:43 smithi194 ceph-mon[45450]: fsmap 3 up:standby 2024-01-24T14:24:43.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:43 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-01-24T14:24:43.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:43 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/3974323326' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T14:24:43.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:43 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:43.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:43 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:43.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:43 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:43.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:43 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T14:24:43.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:43 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T14:24:43.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:43 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:44.709 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T14:24:44.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:44 smithi019 ceph-mon[40663]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:24:44.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:44 smithi019 ceph-mon[40663]: Deploying daemon mds.e on smithi123 2024-01-24T14:24:44.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:44 smithi019 ceph-mon[40663]: mds.? [v2:172.21.15.19:6836/2302964875,v1:172.21.15.19:6837/2302964875] up:boot 2024-01-24T14:24:44.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:44 smithi019 ceph-mon[40663]: fsmap 4 up:standby 2024-01-24T14:24:44.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:44 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-01-24T14:24:44.800 INFO:journalctl@ceph.mds.e.smithi123.stdout:Jan 24 14:24:44 smithi123 systemd[1]: Starting Ceph mds.e for 654b0c8e-bac3-11ee-95b1-87774f69a715... 2024-01-24T14:24:44.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:44 smithi123 ceph-mon[44966]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:24:44.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:44 smithi123 ceph-mon[44966]: Deploying daemon mds.e on smithi123 2024-01-24T14:24:44.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:44 smithi123 ceph-mon[44966]: mds.? [v2:172.21.15.19:6836/2302964875,v1:172.21.15.19:6837/2302964875] up:boot 2024-01-24T14:24:44.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:44 smithi123 ceph-mon[44966]: fsmap 4 up:standby 2024-01-24T14:24:44.801 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:44 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-01-24T14:24:44.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:44 smithi194 ceph-mon[45450]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:24:44.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:44 smithi194 ceph-mon[45450]: Deploying daemon mds.e on smithi123 2024-01-24T14:24:44.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:44 smithi194 ceph-mon[45450]: mds.? [v2:172.21.15.19:6836/2302964875,v1:172.21.15.19:6837/2302964875] up:boot 2024-01-24T14:24:44.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:44 smithi194 ceph-mon[45450]: fsmap 4 up:standby 2024-01-24T14:24:44.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:44 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-01-24T14:24:44.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:44.861+0000 7f6bf9c5f700 1 -- 172.21.15.19:0/1029982970 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6bf4101f50 msgr2=0x7f6bf41023b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:44.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:44.861+0000 7f6bf9c5f700 1 --2- 172.21.15.19:0/1029982970 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6bf4101f50 0x7f6bf41023b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6be8009920 tx=0x7f6be8009c30 comp rx=0 tx=0).stop 2024-01-24T14:24:44.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:44.861+0000 7f6bf9c5f700 1 -- 172.21.15.19:0/1029982970 shutdown_connections 2024-01-24T14:24:44.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:44.861+0000 7f6bf9c5f700 1 --2- 172.21.15.19:0/1029982970 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6bf41028f0 0x7f6bf41091a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:44.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:44.861+0000 7f6bf9c5f700 1 --2- 172.21.15.19:0/1029982970 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6bf4100d60 0x7f6bf4101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:44.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:44.861+0000 7f6bf9c5f700 1 --2- 172.21.15.19:0/1029982970 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6bf4101f50 0x7f6bf41023b0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:44.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:44.861+0000 7f6bf9c5f700 1 -- 172.21.15.19:0/1029982970 >> 172.21.15.19:0/1029982970 conn(0x7f6bf40fc2b0 msgr2=0x7f6bf40fe710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:24:44.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:44.861+0000 7f6bf9c5f700 1 -- 172.21.15.19:0/1029982970 shutdown_connections 2024-01-24T14:24:44.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:44.861+0000 7f6bf9c5f700 1 -- 172.21.15.19:0/1029982970 wait complete. 2024-01-24T14:24:44.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:44.862+0000 7f6bf9c5f700 1 Processor -- start 2024-01-24T14:24:44.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:44.862+0000 7f6bf9c5f700 1 -- start start 2024-01-24T14:24:44.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:44.862+0000 7f6bf9c5f700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6bf4100d60 0x7f6bf4196a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:44.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:44.862+0000 7f6bf9c5f700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6bf4101f50 0x7f6bf4196f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:44.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:44.862+0000 7f6bf9c5f700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6bf41028f0 0x7f6bf419e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:44.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:44.862+0000 7f6bf9c5f700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf419e5d0 con 0x7f6bf4100d60 2024-01-24T14:24:44.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:44.862+0000 7f6bf9c5f700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf419e710 con 0x7f6bf4101f50 2024-01-24T14:24:44.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:44.862+0000 7f6bf9c5f700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bf419e8a0 con 0x7f6bf41028f0 2024-01-24T14:24:44.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:44.862+0000 7f6bf37fe700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6bf4100d60 0x7f6bf4196a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:44.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:44.862+0000 7f6bf37fe700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6bf4100d60 0x7f6bf4196a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:51734/0 (socket says 172.21.15.19:51734) 2024-01-24T14:24:44.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:44.862+0000 7f6bf37fe700 1 -- 172.21.15.19:0/2987228363 learned_addr learned my addr 172.21.15.19:0/2987228363 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:24:44.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:44.862+0000 7f6bf2ffd700 1 --2- 172.21.15.19:0/2987228363 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6bf4101f50 0x7f6bf4196f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:44.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:44.863+0000 7f6bf3fff700 1 --2- 172.21.15.19:0/2987228363 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6bf41028f0 0x7f6bf419e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:44.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:44.863+0000 7f6bf37fe700 1 -- 172.21.15.19:0/2987228363 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6bf41028f0 msgr2=0x7f6bf419e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:44.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:44.863+0000 7f6bf37fe700 1 --2- 172.21.15.19:0/2987228363 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6bf41028f0 0x7f6bf419e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:44.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:44.863+0000 7f6bf37fe700 1 -- 172.21.15.19:0/2987228363 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6bf4101f50 msgr2=0x7f6bf4196f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:44.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:44.863+0000 7f6bf37fe700 1 --2- 172.21.15.19:0/2987228363 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6bf4101f50 0x7f6bf4196f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:44.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:44.863+0000 7f6bf37fe700 1 -- 172.21.15.19:0/2987228363 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be40095f0 con 0x7f6bf4100d60 2024-01-24T14:24:44.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:44.863+0000 7f6bf3fff700 1 --2- 172.21.15.19:0/2987228363 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6bf41028f0 0x7f6bf419e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T14:24:44.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:44.863+0000 7f6bf37fe700 1 --2- 172.21.15.19:0/2987228363 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6bf4100d60 0x7f6bf4196a50 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6be400ebf0 tx=0x7f6be400ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:24:44.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:44.863+0000 7f6bf0ff9700 1 -- 172.21.15.19:0/2987228363 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be4009a10 con 0x7f6bf4100d60 2024-01-24T14:24:44.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:44.863+0000 7f6bf0ff9700 1 -- 172.21.15.19:0/2987228363 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6be400cd40 con 0x7f6bf4100d60 2024-01-24T14:24:44.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:44.863+0000 7f6bf0ff9700 1 -- 172.21.15.19:0/2987228363 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be4018820 con 0x7f6bf4100d60 2024-01-24T14:24:44.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:44.863+0000 7f6bf9c5f700 1 -- 172.21.15.19:0/2987228363 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6be8009520 con 0x7f6bf4100d60 2024-01-24T14:24:44.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:44.863+0000 7f6bf9c5f700 1 -- 172.21.15.19:0/2987228363 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bf419eeb0 con 0x7f6bf4100d60 2024-01-24T14:24:44.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:44.864+0000 7f6bf9c5f700 1 -- 172.21.15.19:0/2987228363 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bf41082f0 con 0x7f6bf4100d60 2024-01-24T14:24:44.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:44.865+0000 7f6bf0ff9700 1 -- 172.21.15.19:0/2987228363 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f6be4018980 con 0x7f6bf4100d60 2024-01-24T14:24:44.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:44.866+0000 7f6bf0ff9700 1 --2- 172.21.15.19:0/2987228363 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f6bdc0a0570 0x7f6bdc0a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:44.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:44.866+0000 7f6bf0ff9700 1 -- 172.21.15.19:0/2987228363 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f6be4014070 con 0x7f6bf4100d60 2024-01-24T14:24:44.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:44.866+0000 7f6bf2ffd700 1 --2- 172.21.15.19:0/2987228363 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f6bdc0a0570 0x7f6bdc0a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:44.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:44.867+0000 7f6bf2ffd700 1 --2- 172.21.15.19:0/2987228363 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f6bdc0a0570 0x7f6bdc0a2a30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6be800fa40 tx=0x7f6be8019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:24:44.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:44.870+0000 7f6bf0ff9700 1 -- 172.21.15.19:0/2987228363 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f6be40962f0 con 0x7f6bf4100d60 2024-01-24T14:24:45.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:45.145+0000 7f6bf9c5f700 1 -- 172.21.15.19:0/2987228363 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6bf402cc70 con 0x7f6bf4100d60 2024-01-24T14:24:45.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:45.146+0000 7f6bf0ff9700 1 -- 172.21.15.19:0/2987228363 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3534 (secure 0 0 0) 0x7f6be4099910 con 0x7f6bf4100d60 2024-01-24T14:24:45.147 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:24:45.147 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6835/1060583512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6834","nonce":1060583512},{"type":"v1","addr":"172.21.15.19:6835","nonce":1060583512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6837/2302964875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6836","nonce":2302964875},{"type":"v1","addr":"172.21.15.19:6837","nonce":2302964875}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2647625404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2647625404},{"type":"v1","addr":"172.21.15.194:6833","nonce":2647625404}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/3396650565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3396650565},{"type":"v1","addr":"172.21.15.123:6833","nonce":3396650565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2024-01-24T14:24:45.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:45.148+0000 7f6bf9c5f700 1 -- 172.21.15.19:0/2987228363 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f6bdc0a0570 msgr2=0x7f6bdc0a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:45.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:45.148+0000 7f6bf9c5f700 1 --2- 172.21.15.19:0/2987228363 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f6bdc0a0570 0x7f6bdc0a2a30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6be800fa40 tx=0x7f6be8019040 comp rx=0 tx=0).stop 2024-01-24T14:24:45.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:45.148+0000 7f6bf9c5f700 1 -- 172.21.15.19:0/2987228363 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6bf4100d60 msgr2=0x7f6bf4196a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:45.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:45.148+0000 7f6bf9c5f700 1 --2- 172.21.15.19:0/2987228363 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6bf4100d60 0x7f6bf4196a50 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6be400ebf0 tx=0x7f6be400ef00 comp rx=0 tx=0).stop 2024-01-24T14:24:45.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:45.148+0000 7f6bf9c5f700 1 -- 172.21.15.19:0/2987228363 shutdown_connections 2024-01-24T14:24:45.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:45.148+0000 7f6bf9c5f700 1 --2- 172.21.15.19:0/2987228363 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f6bdc0a0570 0x7f6bdc0a2a30 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:45.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:45.148+0000 7f6bf9c5f700 1 --2- 172.21.15.19:0/2987228363 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6bf4100d60 0x7f6bf4196a50 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:45.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:45.148+0000 7f6bf9c5f700 1 --2- 172.21.15.19:0/2987228363 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6bf41028f0 0x7f6bf419e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:45.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:45.148+0000 7f6bf9c5f700 1 --2- 172.21.15.19:0/2987228363 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6bf4101f50 0x7f6bf4196f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:45.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:45.148+0000 7f6bf9c5f700 1 -- 172.21.15.19:0/2987228363 >> 172.21.15.19:0/2987228363 conn(0x7f6bf40fc2b0 msgr2=0x7f6bf4105430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:24:45.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:45.149+0000 7f6bf9c5f700 1 -- 172.21.15.19:0/2987228363 shutdown_connections 2024-01-24T14:24:45.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:45.149+0000 7f6bf9c5f700 1 -- 172.21.15.19:0/2987228363 wait complete. 2024-01-24T14:24:45.151 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 5 2024-01-24T14:24:45.299 INFO:journalctl@ceph.mds.e.smithi123.stdout:Jan 24 14:24:44 smithi123 ceph-mds[80700]: -- 172.21.15.123:0/3900512593 <== mon.2 v2:172.21.15.123:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56050bd3b1e0 con 0x56050bd10c00 2024-01-24T14:24:45.299 INFO:journalctl@ceph.mds.e.smithi123.stdout:Jan 24 14:24:44 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-e[80677]: starting mds.e at 2024-01-24T14:24:45.299 INFO:journalctl@ceph.mds.e.smithi123.stdout:Jan 24 14:24:45 smithi123 bash[80662]: 010b823b421c0fa630371e7c84dfde805e10873c32344f41c63d216b7b1cdd4c 2024-01-24T14:24:45.299 INFO:journalctl@ceph.mds.e.smithi123.stdout:Jan 24 14:24:45 smithi123 systemd[1]: Started Ceph mds.e for 654b0c8e-bac3-11ee-95b1-87774f69a715. 2024-01-24T14:24:46.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:46 smithi123 ceph-mon[44966]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:24:46.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:46 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:46.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:46 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:46.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:46 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:46.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:46 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T14:24:46.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:46 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T14:24:46.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:46 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:46.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:46 smithi123 ceph-mon[44966]: Deploying daemon mds.f on smithi194 2024-01-24T14:24:46.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:46 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/2987228363' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T14:24:46.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:46 smithi123 ceph-mon[44966]: mds.? [v2:172.21.15.123:6834/2631676083,v1:172.21.15.123:6835/2631676083] up:boot 2024-01-24T14:24:46.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:46 smithi123 ceph-mon[44966]: fsmap 5 up:standby 2024-01-24T14:24:46.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:46 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-01-24T14:24:46.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:46 smithi194 ceph-mon[45450]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:24:46.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:46 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:46.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:46 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:46.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:46 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:46.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:46 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T14:24:46.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:46 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T14:24:46.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:46 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:46.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:46 smithi194 ceph-mon[45450]: Deploying daemon mds.f on smithi194 2024-01-24T14:24:46.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:46 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/2987228363' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T14:24:46.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:46 smithi194 ceph-mon[45450]: mds.? [v2:172.21.15.123:6834/2631676083,v1:172.21.15.123:6835/2631676083] up:boot 2024-01-24T14:24:46.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:46 smithi194 ceph-mon[45450]: fsmap 5 up:standby 2024-01-24T14:24:46.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:46 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-01-24T14:24:46.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:46 smithi019 ceph-mon[40663]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:24:46.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:46 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:46.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:46 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:46.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:46 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:46.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:46 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T14:24:46.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:46 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T14:24:46.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:46 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:46.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:46 smithi019 ceph-mon[40663]: Deploying daemon mds.f on smithi194 2024-01-24T14:24:46.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:46 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/2987228363' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T14:24:46.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:46 smithi019 ceph-mon[40663]: mds.? [v2:172.21.15.123:6834/2631676083,v1:172.21.15.123:6835/2631676083] up:boot 2024-01-24T14:24:46.487 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:46 smithi019 ceph-mon[40663]: fsmap 5 up:standby 2024-01-24T14:24:46.487 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:46 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-01-24T14:24:47.094 INFO:journalctl@ceph.mds.f.smithi194.stdout:Jan 24 14:24:46 smithi194 bash[82391]: 99f859c61df5d03b9b7ff7dc8ceb4b442457483ecbb01eaf7eeb6174b63af91c 2024-01-24T14:24:47.094 INFO:journalctl@ceph.mds.f.smithi194.stdout:Jan 24 14:24:46 smithi194 systemd[1]: Started Ceph mds.f for 654b0c8e-bac3-11ee-95b1-87774f69a715. 2024-01-24T14:24:47.167 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T14:24:47.313 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.310+0000 7f016f416700 1 -- 172.21.15.19:0/4281098011 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f01681017e0 msgr2=0x7f0168108090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:47.314 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.310+0000 7f016f416700 1 --2- 172.21.15.19:0/4281098011 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f01681017e0 0x7f0168108090 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f016000b0d0 tx=0x7f016000b3e0 comp rx=0 tx=0).stop 2024-01-24T14:24:47.314 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.310+0000 7f016f416700 1 -- 172.21.15.19:0/4281098011 shutdown_connections 2024-01-24T14:24:47.314 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.310+0000 7f016f416700 1 --2- 172.21.15.19:0/4281098011 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f01681017e0 0x7f0168108090 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:47.314 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.310+0000 7f016f416700 1 --2- 172.21.15.19:0/4281098011 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f01680ffc50 0x7f0168100070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:47.314 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.310+0000 7f016f416700 1 --2- 172.21.15.19:0/4281098011 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0168100e40 0x7f01681012a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:47.314 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.310+0000 7f016f416700 1 -- 172.21.15.19:0/4281098011 >> 172.21.15.19:0/4281098011 conn(0x7f01680fb1a0 msgr2=0x7f01680fd600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:24:47.314 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.310+0000 7f016f416700 1 -- 172.21.15.19:0/4281098011 shutdown_connections 2024-01-24T14:24:47.314 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.310+0000 7f016f416700 1 -- 172.21.15.19:0/4281098011 wait complete. 2024-01-24T14:24:47.314 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.311+0000 7f016f416700 1 Processor -- start 2024-01-24T14:24:47.314 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.311+0000 7f016f416700 1 -- start start 2024-01-24T14:24:47.314 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.311+0000 7f016f416700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f01680ffc50 0x7f0168196ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:47.314 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.311+0000 7f016f416700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0168100e40 0x7f0168196ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:47.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.311+0000 7f016f416700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01681017e0 0x7f016819e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:47.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.311+0000 7f016f416700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f016819e630 con 0x7f0168100e40 2024-01-24T14:24:47.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.311+0000 7f016f416700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f016819e770 con 0x7f01681017e0 2024-01-24T14:24:47.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.311+0000 7f016f416700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f016819e900 con 0x7f01680ffc50 2024-01-24T14:24:47.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.311+0000 7f016c9b1700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0168100e40 0x7f0168196ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:47.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.311+0000 7f016c9b1700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0168100e40 0x7f0168196ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:51748/0 (socket says 172.21.15.19:51748) 2024-01-24T14:24:47.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.311+0000 7f016c9b1700 1 -- 172.21.15.19:0/830226037 learned_addr learned my addr 172.21.15.19:0/830226037 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:24:47.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.311+0000 7f016d9b3700 1 --2- 172.21.15.19:0/830226037 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01681017e0 0x7f016819e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:47.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.311+0000 7f016d1b2700 1 --2- 172.21.15.19:0/830226037 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f01680ffc50 0x7f0168196ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:47.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.312+0000 7f016c9b1700 1 -- 172.21.15.19:0/830226037 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f01680ffc50 msgr2=0x7f0168196ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:47.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.312+0000 7f016c9b1700 1 --2- 172.21.15.19:0/830226037 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f01680ffc50 0x7f0168196ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:47.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.312+0000 7f016c9b1700 1 -- 172.21.15.19:0/830226037 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01681017e0 msgr2=0x7f016819e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:47.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.312+0000 7f016c9b1700 1 --2- 172.21.15.19:0/830226037 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01681017e0 0x7f016819e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:47.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.312+0000 7f016c9b1700 1 -- 172.21.15.19:0/830226037 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0160009d00 con 0x7f0168100e40 2024-01-24T14:24:47.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.312+0000 7f016d1b2700 1 --2- 172.21.15.19:0/830226037 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f01680ffc50 0x7f0168196ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T14:24:47.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.312+0000 7f016d9b3700 1 --2- 172.21.15.19:0/830226037 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01681017e0 0x7f016819e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T14:24:47.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.312+0000 7f016c9b1700 1 --2- 172.21.15.19:0/830226037 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0168100e40 0x7f0168196ff0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f015800e9a0 tx=0x7f015800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:24:47.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.312+0000 7f015e7fc700 1 -- 172.21.15.19:0/830226037 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0158009790 con 0x7f0168100e40 2024-01-24T14:24:47.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.312+0000 7f016f416700 1 -- 172.21.15.19:0/830226037 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f016819ebe0 con 0x7f0168100e40 2024-01-24T14:24:47.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.312+0000 7f015e7fc700 1 -- 172.21.15.19:0/830226037 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0158010430 con 0x7f0168100e40 2024-01-24T14:24:47.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.312+0000 7f015e7fc700 1 -- 172.21.15.19:0/830226037 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0158018680 con 0x7f0168100e40 2024-01-24T14:24:47.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.312+0000 7f016f416700 1 -- 172.21.15.19:0/830226037 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f016819f0f0 con 0x7f0168100e40 2024-01-24T14:24:47.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.313+0000 7f016f416700 1 -- 172.21.15.19:0/830226037 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f016810b470 con 0x7f0168100e40 2024-01-24T14:24:47.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.314+0000 7f015e7fc700 1 -- 172.21.15.19:0/830226037 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f01580105a0 con 0x7f0168100e40 2024-01-24T14:24:47.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.315+0000 7f015e7fc700 1 --2- 172.21.15.19:0/830226037 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f01540a0570 0x7f01540a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:47.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.315+0000 7f016d1b2700 1 --2- 172.21.15.19:0/830226037 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f01540a0570 0x7f01540a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:47.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.315+0000 7f015e7fc700 1 -- 172.21.15.19:0/830226037 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f0158014070 con 0x7f0168100e40 2024-01-24T14:24:47.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.315+0000 7f016d1b2700 1 --2- 172.21.15.19:0/830226037 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f01540a0570 0x7f01540a2a30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0168100ca0 tx=0x7f0164009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:24:47.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.319+0000 7f015e7fc700 1 -- 172.21.15.19:0/830226037 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f0158095fc0 con 0x7f0168100e40 2024-01-24T14:24:47.579 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.579+0000 7f016f416700 1 -- 172.21.15.19:0/830226037 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f016810b700 con 0x7f0168100e40 2024-01-24T14:24:47.579 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.579+0000 7f015e7fc700 1 -- 172.21.15.19:0/830226037 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5051 (secure 0 0 0) 0x7f016810b700 con 0x7f0168100e40 2024-01-24T14:24:47.579 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:24:47.580 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6835/1060583512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6834","nonce":1060583512},{"type":"v1","addr":"172.21.15.19:6835","nonce":1060583512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6837/2302964875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6836","nonce":2302964875},{"type":"v1","addr":"172.21.15.19:6837","nonce":2302964875}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2647625404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2647625404},{"type":"v1","addr":"172.21.15.194:6833","nonce":2647625404}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/3396650565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3396650565},{"type":"v1","addr":"172.21.15.123:6833","nonce":3396650565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24485,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2631676083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2631676083},{"type":"v1","addr":"172.21.15.123:6835","nonce":2631676083}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6835/456696823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":456696823},{"type":"v1","addr":"172.21.15.194:6835","nonce":456696823}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[]} 2024-01-24T14:24:47.582 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.581+0000 7f016f416700 1 -- 172.21.15.19:0/830226037 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f01540a0570 msgr2=0x7f01540a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:47.582 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.581+0000 7f016f416700 1 --2- 172.21.15.19:0/830226037 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f01540a0570 0x7f01540a2a30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0168100ca0 tx=0x7f0164009260 comp rx=0 tx=0).stop 2024-01-24T14:24:47.582 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.582+0000 7f016f416700 1 -- 172.21.15.19:0/830226037 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0168100e40 msgr2=0x7f0168196ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:47.582 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.582+0000 7f016f416700 1 --2- 172.21.15.19:0/830226037 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0168100e40 0x7f0168196ff0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f015800e9a0 tx=0x7f015800ecb0 comp rx=0 tx=0).stop 2024-01-24T14:24:47.582 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.582+0000 7f016f416700 1 -- 172.21.15.19:0/830226037 shutdown_connections 2024-01-24T14:24:47.582 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.582+0000 7f016f416700 1 --2- 172.21.15.19:0/830226037 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01681017e0 0x7f016819e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:47.582 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.582+0000 7f016f416700 1 --2- 172.21.15.19:0/830226037 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f01680ffc50 0x7f0168196ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:47.582 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.582+0000 7f016f416700 1 --2- 172.21.15.19:0/830226037 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f01540a0570 0x7f01540a2a30 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:47.582 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.582+0000 7f016f416700 1 --2- 172.21.15.19:0/830226037 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0168100e40 0x7f0168196ff0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:47.583 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.582+0000 7f016f416700 1 -- 172.21.15.19:0/830226037 >> 172.21.15.19:0/830226037 conn(0x7f01680fb1a0 msgr2=0x7f0168069140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:24:47.583 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.582+0000 7f016f416700 1 -- 172.21.15.19:0/830226037 shutdown_connections 2024-01-24T14:24:47.583 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:47.582+0000 7f016f416700 1 -- 172.21.15.19:0/830226037 wait complete. 2024-01-24T14:24:47.583 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 7 2024-01-24T14:24:47.791 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:47 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:47.791 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:47 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:47.791 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:47 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:47.791 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:47 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T14:24:47.791 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:47 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T14:24:47.791 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:47 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:47.791 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:47 smithi194 ceph-mon[45450]: Deploying daemon mds.c on smithi194 2024-01-24T14:24:47.792 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:47 smithi194 ceph-mon[45450]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:24:47.792 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:47 smithi194 ceph-mon[45450]: mds.? [v2:172.21.15.194:6834/456696823,v1:172.21.15.194:6835/456696823] up:boot 2024-01-24T14:24:47.792 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:47 smithi194 ceph-mon[45450]: fsmap 6 up:standby 2024-01-24T14:24:47.792 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:47 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-01-24T14:24:47.792 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:47 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/830226037' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T14:24:47.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:47 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:47.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:47 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:47.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:47 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:47.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:47 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T14:24:47.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:47 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T14:24:47.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:47 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:47.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:47 smithi019 ceph-mon[40663]: Deploying daemon mds.c on smithi194 2024-01-24T14:24:47.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:47 smithi019 ceph-mon[40663]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:24:47.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:47 smithi019 ceph-mon[40663]: mds.? [v2:172.21.15.194:6834/456696823,v1:172.21.15.194:6835/456696823] up:boot 2024-01-24T14:24:47.987 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:47 smithi019 ceph-mon[40663]: fsmap 6 up:standby 2024-01-24T14:24:47.987 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:47 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-01-24T14:24:47.987 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:47 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/830226037' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T14:24:48.042 INFO:journalctl@ceph.mds.c.smithi194.stdout:Jan 24 14:24:48 smithi194 systemd[1]: Starting Ceph mds.c for 654b0c8e-bac3-11ee-95b1-87774f69a715... 2024-01-24T14:24:48.049 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:47 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:48.049 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:47 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:48.049 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:47 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:48.049 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:47 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T14:24:48.050 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:47 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T14:24:48.050 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:47 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:48.050 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:47 smithi123 ceph-mon[44966]: Deploying daemon mds.c on smithi194 2024-01-24T14:24:48.050 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:47 smithi123 ceph-mon[44966]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:24:48.050 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:47 smithi123 ceph-mon[44966]: mds.? [v2:172.21.15.194:6834/456696823,v1:172.21.15.194:6835/456696823] up:boot 2024-01-24T14:24:48.050 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:47 smithi123 ceph-mon[44966]: fsmap 6 up:standby 2024-01-24T14:24:48.050 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:47 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-01-24T14:24:48.050 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:47 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/830226037' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T14:24:48.844 INFO:journalctl@ceph.mds.c.smithi194.stdout:Jan 24 14:24:48 smithi194 ceph-mds[82769]: -- 172.21.15.194:0/2291206881 <== mon.1 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55cef99491e0 con 0x55cef991ec00 2024-01-24T14:24:48.844 INFO:journalctl@ceph.mds.c.smithi194.stdout:Jan 24 14:24:48 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-c[82746]: starting mds.c at 2024-01-24T14:24:48.844 INFO:journalctl@ceph.mds.c.smithi194.stdout:Jan 24 14:24:48 smithi194 bash[82731]: 82de9ebbc6278c1b7b38e4aa889411b8ea77a1f8329de84474349991c631a511 2024-01-24T14:24:48.844 INFO:journalctl@ceph.mds.c.smithi194.stdout:Jan 24 14:24:48 smithi194 systemd[1]: Started Ceph mds.c for 654b0c8e-bac3-11ee-95b1-87774f69a715. 2024-01-24T14:24:49.596 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T14:24:49.677 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:49 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:49.677 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:49 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:49.677 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:49 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:49.678 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:49 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T14:24:49.678 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:49 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T14:24:49.678 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:49 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:49.678 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:49 smithi123 ceph-mon[44966]: Deploying daemon mds.h on smithi123 2024-01-24T14:24:49.678 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:49 smithi123 ceph-mon[44966]: mds.? [v2:172.21.15.194:6836/2623846234,v1:172.21.15.194:6837/2623846234] up:boot 2024-01-24T14:24:49.678 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:49 smithi123 ceph-mon[44966]: fsmap 7 up:standby 2024-01-24T14:24:49.678 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:49 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-01-24T14:24:49.678 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:49 smithi123 ceph-mon[44966]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:24:49.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:49 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:49.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:49 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:49.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:49 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:49.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:49 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T14:24:49.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:49 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T14:24:49.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:49 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:49.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:49 smithi019 ceph-mon[40663]: Deploying daemon mds.h on smithi123 2024-01-24T14:24:49.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:49 smithi019 ceph-mon[40663]: mds.? [v2:172.21.15.194:6836/2623846234,v1:172.21.15.194:6837/2623846234] up:boot 2024-01-24T14:24:49.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:49 smithi019 ceph-mon[40663]: fsmap 7 up:standby 2024-01-24T14:24:49.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:49 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-01-24T14:24:49.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:49 smithi019 ceph-mon[40663]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:24:49.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:49.742+0000 7fbeee499700 1 -- 172.21.15.19:0/2385195854 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbee80feff0 msgr2=0x7fbee80ff410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:49.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:49.742+0000 7fbeee499700 1 --2- 172.21.15.19:0/2385195854 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbee80feff0 0x7fbee80ff410 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fbed8009990 tx=0x7fbed8009ca0 comp rx=0 tx=0).stop 2024-01-24T14:24:49.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:49.742+0000 7fbeee499700 1 -- 172.21.15.19:0/2385195854 shutdown_connections 2024-01-24T14:24:49.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:49.742+0000 7fbeee499700 1 --2- 172.21.15.19:0/2385195854 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbee80feff0 0x7fbee80ff410 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:49.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:49.742+0000 7fbeee499700 1 --2- 172.21.15.19:0/2385195854 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbee8102960 0x7fbee8109210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:49.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:49.742+0000 7fbeee499700 1 --2- 172.21.15.19:0/2385195854 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbee8101fc0 0x7fbee8102420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:49.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:49.742+0000 7fbeee499700 1 -- 172.21.15.19:0/2385195854 >> 172.21.15.19:0/2385195854 conn(0x7fbee80fa900 msgr2=0x7fbee80fcd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:24:49.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:49.742+0000 7fbeee499700 1 -- 172.21.15.19:0/2385195854 shutdown_connections 2024-01-24T14:24:49.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:49.742+0000 7fbeee499700 1 -- 172.21.15.19:0/2385195854 wait complete. 2024-01-24T14:24:49.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:49.742+0000 7fbeee499700 1 Processor -- start 2024-01-24T14:24:49.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:49.743+0000 7fbeee499700 1 -- start start 2024-01-24T14:24:49.745 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:49.743+0000 7fbeee499700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbee80feff0 0x7fbee8196aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:49.745 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:49.743+0000 7fbeee499700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbee8101fc0 0x7fbee8196fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:49.745 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:49.743+0000 7fbeee499700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbee8102960 0x7fbee819e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:49.745 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:49.743+0000 7fbeee499700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbee819e620 con 0x7fbee8102960 2024-01-24T14:24:49.745 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:49.743+0000 7fbeee499700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbee819e760 con 0x7fbee8101fc0 2024-01-24T14:24:49.745 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:49.743+0000 7fbeee499700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbee819e8f0 con 0x7fbee80feff0 2024-01-24T14:24:49.745 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:49.743+0000 7fbeeca36700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbee8102960 0x7fbee819e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:49.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:49.743+0000 7fbee77fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbee8101fc0 0x7fbee8196fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:49.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:49.743+0000 7fbee77fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbee8101fc0 0x7fbee8196fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.19:37214/0 (socket says 172.21.15.19:37214) 2024-01-24T14:24:49.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:49.743+0000 7fbeeca36700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbee8102960 0x7fbee819e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:51770/0 (socket says 172.21.15.19:51770) 2024-01-24T14:24:49.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:49.743+0000 7fbee77fe700 1 -- 172.21.15.19:0/3932109469 learned_addr learned my addr 172.21.15.19:0/3932109469 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:24:49.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:49.743+0000 7fbee7fff700 1 --2- 172.21.15.19:0/3932109469 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbee80feff0 0x7fbee8196aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:49.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:49.743+0000 7fbeeca36700 1 -- 172.21.15.19:0/3932109469 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbee80feff0 msgr2=0x7fbee8196aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:49.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:49.743+0000 7fbeeca36700 1 --2- 172.21.15.19:0/3932109469 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbee80feff0 0x7fbee8196aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:49.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:49.743+0000 7fbeeca36700 1 -- 172.21.15.19:0/3932109469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbee8101fc0 msgr2=0x7fbee8196fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:49.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:49.743+0000 7fbeeca36700 1 --2- 172.21.15.19:0/3932109469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbee8101fc0 0x7fbee8196fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:49.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:49.744+0000 7fbeeca36700 1 -- 172.21.15.19:0/3932109469 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbed80095f0 con 0x7fbee8102960 2024-01-24T14:24:49.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:49.744+0000 7fbeeca36700 1 --2- 172.21.15.19:0/3932109469 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbee8102960 0x7fbee819e050 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fbee000be20 tx=0x7fbee0009ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:24:49.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:49.744+0000 7fbee57fa700 1 -- 172.21.15.19:0/3932109469 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee000cb90 con 0x7fbee8102960 2024-01-24T14:24:49.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:49.744+0000 7fbee57fa700 1 -- 172.21.15.19:0/3932109469 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbee000ccf0 con 0x7fbee8102960 2024-01-24T14:24:49.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:49.744+0000 7fbeee499700 1 -- 172.21.15.19:0/3932109469 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbee819ebd0 con 0x7fbee8102960 2024-01-24T14:24:49.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:49.744+0000 7fbee57fa700 1 -- 172.21.15.19:0/3932109469 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee0012670 con 0x7fbee8102960 2024-01-24T14:24:49.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:49.744+0000 7fbeee499700 1 -- 172.21.15.19:0/3932109469 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbee819f090 con 0x7fbee8102960 2024-01-24T14:24:49.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:49.745+0000 7fbeee499700 1 -- 172.21.15.19:0/3932109469 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbee810c600 con 0x7fbee8102960 2024-01-24T14:24:49.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:49.746+0000 7fbee57fa700 1 -- 172.21.15.19:0/3932109469 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7fbee0014440 con 0x7fbee8102960 2024-01-24T14:24:49.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:49.747+0000 7fbee57fa700 1 --2- 172.21.15.19:0/3932109469 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fbed40a0570 0x7fbed40a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:49.748 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:49.747+0000 7fbee7fff700 1 --2- 172.21.15.19:0/3932109469 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fbed40a0570 0x7fbed40a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:49.748 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:49.747+0000 7fbee57fa700 1 -- 172.21.15.19:0/3932109469 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fbee0006e40 con 0x7fbee8102960 2024-01-24T14:24:49.748 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:49.747+0000 7fbee7fff700 1 --2- 172.21.15.19:0/3932109469 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fbed40a0570 0x7fbed40a2a30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fbed800b5c0 tx=0x7fbed8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:24:49.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:49.751+0000 7fbee57fa700 1 -- 172.21.15.19:0/3932109469 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fbee0094d80 con 0x7fbee8102960 2024-01-24T14:24:49.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:49 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:49.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:49 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:49.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:49 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:49.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:49 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T14:24:49.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:49 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T14:24:49.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:49 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:49.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:49 smithi194 ceph-mon[45450]: Deploying daemon mds.h on smithi123 2024-01-24T14:24:49.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:49 smithi194 ceph-mon[45450]: mds.? [v2:172.21.15.194:6836/2623846234,v1:172.21.15.194:6837/2623846234] up:boot 2024-01-24T14:24:49.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:49 smithi194 ceph-mon[45450]: fsmap 7 up:standby 2024-01-24T14:24:49.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:49 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-01-24T14:24:49.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:49 smithi194 ceph-mon[45450]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:24:49.929 INFO:journalctl@ceph.mds.h.smithi123.stdout:Jan 24 14:24:49 smithi123 systemd[1]: Starting Ceph mds.h for 654b0c8e-bac3-11ee-95b1-87774f69a715... 2024-01-24T14:24:50.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:50.014+0000 7fbeee499700 1 -- 172.21.15.19:0/3932109469 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbee8107d10 con 0x7fbee8102960 2024-01-24T14:24:50.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:50.014+0000 7fbee57fa700 1 -- 172.21.15.19:0/3932109469 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+5811 (secure 0 0 0) 0x7fbee000ce60 con 0x7fbee8102960 2024-01-24T14:24:50.014 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:24:50.014 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6835/1060583512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6834","nonce":1060583512},{"type":"v1","addr":"172.21.15.19:6835","nonce":1060583512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6837/2302964875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6836","nonce":2302964875},{"type":"v1","addr":"172.21.15.19:6837","nonce":2302964875}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2647625404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2647625404},{"type":"v1","addr":"172.21.15.194:6833","nonce":2647625404}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/3396650565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3396650565},{"type":"v1","addr":"172.21.15.123:6833","nonce":3396650565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24485,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2631676083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2631676083},{"type":"v1","addr":"172.21.15.123:6835","nonce":2631676083}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6835/456696823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":456696823},{"type":"v1","addr":"172.21.15.194:6835","nonce":456696823}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6837/2623846234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":2623846234},{"type":"v1","addr":"172.21.15.194:6837","nonce":2623846234}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[]} 2024-01-24T14:24:50.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:50.016+0000 7fbeee499700 1 -- 172.21.15.19:0/3932109469 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fbed40a0570 msgr2=0x7fbed40a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:50.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:50.016+0000 7fbeee499700 1 --2- 172.21.15.19:0/3932109469 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fbed40a0570 0x7fbed40a2a30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fbed800b5c0 tx=0x7fbed8005fb0 comp rx=0 tx=0).stop 2024-01-24T14:24:50.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:50.016+0000 7fbeee499700 1 -- 172.21.15.19:0/3932109469 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbee8102960 msgr2=0x7fbee819e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:50.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:50.016+0000 7fbeee499700 1 --2- 172.21.15.19:0/3932109469 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbee8102960 0x7fbee819e050 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fbee000be20 tx=0x7fbee0009ed0 comp rx=0 tx=0).stop 2024-01-24T14:24:50.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:50.016+0000 7fbeee499700 1 -- 172.21.15.19:0/3932109469 shutdown_connections 2024-01-24T14:24:50.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:50.016+0000 7fbeee499700 1 --2- 172.21.15.19:0/3932109469 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fbed40a0570 0x7fbed40a2a30 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:50.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:50.016+0000 7fbeee499700 1 --2- 172.21.15.19:0/3932109469 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbee8102960 0x7fbee819e050 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:50.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:50.016+0000 7fbeee499700 1 --2- 172.21.15.19:0/3932109469 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbee80feff0 0x7fbee8196aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:50.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:50.016+0000 7fbeee499700 1 --2- 172.21.15.19:0/3932109469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbee8101fc0 0x7fbee8196fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:50.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:50.016+0000 7fbeee499700 1 -- 172.21.15.19:0/3932109469 >> 172.21.15.19:0/3932109469 conn(0x7fbee80fa900 msgr2=0x7fbee80fcbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:24:50.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:50.017+0000 7fbeee499700 1 -- 172.21.15.19:0/3932109469 shutdown_connections 2024-01-24T14:24:50.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:50.017+0000 7fbeee499700 1 -- 172.21.15.19:0/3932109469 wait complete. 2024-01-24T14:24:50.017 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 8 2024-01-24T14:24:50.799 INFO:journalctl@ceph.mds.h.smithi123.stdout:Jan 24 14:24:50 smithi123 ceph-mds[81041]: -- 172.21.15.123:0/3145694654 <== mon.2 v2:172.21.15.123:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5570e6fad1e0 con 0x5570e6f82c00 2024-01-24T14:24:50.799 INFO:journalctl@ceph.mds.h.smithi123.stdout:Jan 24 14:24:50 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-h[81018]: starting mds.h at 2024-01-24T14:24:50.799 INFO:journalctl@ceph.mds.h.smithi123.stdout:Jan 24 14:24:50 smithi123 bash[81003]: 725cd563eae88e7055bfe98b23a53e4a9a1d30615d19e03baa00bd95bcaaac1d 2024-01-24T14:24:50.800 INFO:journalctl@ceph.mds.h.smithi123.stdout:Jan 24 14:24:50 smithi123 systemd[1]: Started Ceph mds.h for 654b0c8e-bac3-11ee-95b1-87774f69a715. 2024-01-24T14:24:51.071 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:50 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/3932109469' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T14:24:51.071 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:50 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:51.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:50 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:51.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:50 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:51.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:50 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T14:24:51.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:50 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T14:24:51.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:50 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:51.072 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:50 smithi194 ceph-mon[45450]: Deploying daemon mds.i on smithi194 2024-01-24T14:24:51.235 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:50 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/3932109469' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T14:24:51.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:50 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:51.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:50 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:51.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:50 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:51.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:50 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T14:24:51.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:50 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T14:24:51.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:50 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:51.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:50 smithi019 ceph-mon[40663]: Deploying daemon mds.i on smithi194 2024-01-24T14:24:51.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:50 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/3932109469' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T14:24:51.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:50 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:51.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:50 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:51.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:50 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:51.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:50 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T14:24:51.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:50 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T14:24:51.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:50 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:51.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:50 smithi123 ceph-mon[44966]: Deploying daemon mds.i on smithi194 2024-01-24T14:24:51.973 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:51 smithi194 ceph-mon[45450]: mds.? [v2:172.21.15.123:6836/491309336,v1:172.21.15.123:6837/491309336] up:boot 2024-01-24T14:24:51.974 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:51 smithi194 ceph-mon[45450]: fsmap 8 up:standby 2024-01-24T14:24:51.974 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:51 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-01-24T14:24:51.974 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:51 smithi194 ceph-mon[45450]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:24:51.974 INFO:journalctl@ceph.mds.i.smithi194.stdout:Jan 24 14:24:51 smithi194 systemd[1]: Starting Ceph mds.i for 654b0c8e-bac3-11ee-95b1-87774f69a715... 2024-01-24T14:24:52.031 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T14:24:52.183 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.180+0000 7fe52e51e700 1 -- 172.21.15.19:0/875803778 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe528106a00 msgr2=0x7fe528108e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:52.183 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.180+0000 7fe52e51e700 1 --2- 172.21.15.19:0/875803778 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe528106a00 0x7fe528108e70 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe52000b0a0 tx=0x7fe52000b3b0 comp rx=0 tx=0).stop 2024-01-24T14:24:52.183 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.180+0000 7fe52e51e700 1 -- 172.21.15.19:0/875803778 shutdown_connections 2024-01-24T14:24:52.183 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.180+0000 7fe52e51e700 1 --2- 172.21.15.19:0/875803778 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe528106a00 0x7fe528108e70 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:52.184 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.180+0000 7fe52e51e700 1 --2- 172.21.15.19:0/875803778 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5281017a0 0x7fe528103b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:52.184 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.180+0000 7fe52e51e700 1 --2- 172.21.15.19:0/875803778 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe5281040d0 0x7fe5281064c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:52.184 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.180+0000 7fe52e51e700 1 -- 172.21.15.19:0/875803778 >> 172.21.15.19:0/875803778 conn(0x7fe5280fb170 msgr2=0x7fe5280fd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:24:52.184 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.180+0000 7fe52e51e700 1 -- 172.21.15.19:0/875803778 shutdown_connections 2024-01-24T14:24:52.184 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.180+0000 7fe52e51e700 1 -- 172.21.15.19:0/875803778 wait complete. 2024-01-24T14:24:52.184 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.181+0000 7fe52e51e700 1 Processor -- start 2024-01-24T14:24:52.184 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.181+0000 7fe52e51e700 1 -- start start 2024-01-24T14:24:52.184 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.181+0000 7fe52e51e700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5281017a0 0x7fe528196aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:52.184 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.181+0000 7fe52e51e700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe5281040d0 0x7fe528196fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:52.184 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.181+0000 7fe52e51e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe528106a00 0x7fe52819e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:52.184 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.181+0000 7fe52e51e700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52819e620 con 0x7fe5281040d0 2024-01-24T14:24:52.184 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.181+0000 7fe52e51e700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52819e760 con 0x7fe5281017a0 2024-01-24T14:24:52.184 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.181+0000 7fe52e51e700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52819e8f0 con 0x7fe528106a00 2024-01-24T14:24:52.185 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.181+0000 7fe5277fe700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe5281040d0 0x7fe528196fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:52.185 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.181+0000 7fe5277fe700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe5281040d0 0x7fe528196fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:54134/0 (socket says 172.21.15.19:54134) 2024-01-24T14:24:52.185 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.181+0000 7fe5277fe700 1 -- 172.21.15.19:0/1263797109 learned_addr learned my addr 172.21.15.19:0/1263797109 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:24:52.185 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.181+0000 7fe527fff700 1 --2- 172.21.15.19:0/1263797109 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5281017a0 0x7fe528196aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:52.185 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.182+0000 7fe52cabb700 1 --2- 172.21.15.19:0/1263797109 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe528106a00 0x7fe52819e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:52.185 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.182+0000 7fe5277fe700 1 -- 172.21.15.19:0/1263797109 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe528106a00 msgr2=0x7fe52819e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:52.185 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.182+0000 7fe5277fe700 1 --2- 172.21.15.19:0/1263797109 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe528106a00 0x7fe52819e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:52.185 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.182+0000 7fe5277fe700 1 -- 172.21.15.19:0/1263797109 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5281017a0 msgr2=0x7fe528196aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:52.185 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.182+0000 7fe5277fe700 1 --2- 172.21.15.19:0/1263797109 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5281017a0 0x7fe528196aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:52.185 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.182+0000 7fe5277fe700 1 -- 172.21.15.19:0/1263797109 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe520009d00 con 0x7fe5281040d0 2024-01-24T14:24:52.185 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.182+0000 7fe5277fe700 1 --2- 172.21.15.19:0/1263797109 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe5281040d0 0x7fe528196fe0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe51c00d950 tx=0x7fe51c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:24:52.185 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.182+0000 7fe5257fa700 1 -- 172.21.15.19:0/1263797109 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51c00f890 con 0x7fe5281040d0 2024-01-24T14:24:52.185 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.182+0000 7fe5257fa700 1 -- 172.21.15.19:0/1263797109 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe51c010460 con 0x7fe5281040d0 2024-01-24T14:24:52.185 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.182+0000 7fe52e51e700 1 -- 172.21.15.19:0/1263797109 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe52819ebd0 con 0x7fe5281040d0 2024-01-24T14:24:52.186 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.182+0000 7fe5257fa700 1 -- 172.21.15.19:0/1263797109 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51c00e610 con 0x7fe5281040d0 2024-01-24T14:24:52.186 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.182+0000 7fe52e51e700 1 -- 172.21.15.19:0/1263797109 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe52819f130 con 0x7fe5281040d0 2024-01-24T14:24:52.186 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.183+0000 7fe52e51e700 1 -- 172.21.15.19:0/1263797109 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5280fccf0 con 0x7fe5281040d0 2024-01-24T14:24:52.186 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.185+0000 7fe5257fa700 1 -- 172.21.15.19:0/1263797109 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7fe51c0108d0 con 0x7fe5281040d0 2024-01-24T14:24:52.187 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.185+0000 7fe5257fa700 1 --2- 172.21.15.19:0/1263797109 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fe5100a05f0 0x7fe5100a2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:52.187 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.185+0000 7fe527fff700 1 --2- 172.21.15.19:0/1263797109 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fe5100a05f0 0x7fe5100a2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:52.187 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.185+0000 7fe5257fa700 1 -- 172.21.15.19:0/1263797109 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe51c0cb190 con 0x7fe5281040d0 2024-01-24T14:24:52.187 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.186+0000 7fe527fff700 1 --2- 172.21.15.19:0/1263797109 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fe5100a05f0 0x7fe5100a2ab0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe518007c60 tx=0x7fe5180073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:24:52.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.189+0000 7fe5257fa700 1 -- 172.21.15.19:0/1263797109 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fe51c0951d0 con 0x7fe5281040d0 2024-01-24T14:24:52.235 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:51 smithi019 ceph-mon[40663]: mds.? [v2:172.21.15.123:6836/491309336,v1:172.21.15.123:6837/491309336] up:boot 2024-01-24T14:24:52.235 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:51 smithi019 ceph-mon[40663]: fsmap 8 up:standby 2024-01-24T14:24:52.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:51 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-01-24T14:24:52.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:51 smithi019 ceph-mon[40663]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:24:52.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:51 smithi123 ceph-mon[44966]: mds.? [v2:172.21.15.123:6836/491309336,v1:172.21.15.123:6837/491309336] up:boot 2024-01-24T14:24:52.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:51 smithi123 ceph-mon[44966]: fsmap 8 up:standby 2024-01-24T14:24:52.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:51 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-01-24T14:24:52.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:51 smithi123 ceph-mon[44966]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:24:52.344 INFO:journalctl@ceph.mds.i.smithi194.stdout:Jan 24 14:24:51 smithi194 ceph-mds[83109]: -- 172.21.15.194:0/2330656010 <== mon.1 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55775e4c91e0 con 0x55775e49f000 2024-01-24T14:24:52.344 INFO:journalctl@ceph.mds.i.smithi194.stdout:Jan 24 14:24:51 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-i[83086]: starting mds.i at 2024-01-24T14:24:52.344 INFO:journalctl@ceph.mds.i.smithi194.stdout:Jan 24 14:24:52 smithi194 bash[83071]: d5677f20eedcdecbc815c68a54e928a075c99c0eb104f58bb3a1797fc9790743 2024-01-24T14:24:52.344 INFO:journalctl@ceph.mds.i.smithi194.stdout:Jan 24 14:24:52 smithi194 systemd[1]: Started Ceph mds.i for 654b0c8e-bac3-11ee-95b1-87774f69a715. 2024-01-24T14:24:52.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.491+0000 7fe52e51e700 1 -- 172.21.15.19:0/1263797109 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe52802cc70 con 0x7fe5281040d0 2024-01-24T14:24:52.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.491+0000 7fe5257fa700 1 -- 172.21.15.19:0/1263797109 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6568 (secure 0 0 0) 0x7fe51c0987f0 con 0x7fe5281040d0 2024-01-24T14:24:52.491 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:24:52.492 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6835/1060583512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6834","nonce":1060583512},{"type":"v1","addr":"172.21.15.19:6835","nonce":1060583512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6837/2302964875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6836","nonce":2302964875},{"type":"v1","addr":"172.21.15.19:6837","nonce":2302964875}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2647625404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2647625404},{"type":"v1","addr":"172.21.15.194:6833","nonce":2647625404}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/3396650565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3396650565},{"type":"v1","addr":"172.21.15.123:6833","nonce":3396650565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24485,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2631676083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2631676083},{"type":"v1","addr":"172.21.15.123:6835","nonce":2631676083}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6835/456696823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":456696823},{"type":"v1","addr":"172.21.15.194:6835","nonce":456696823}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6837/2623846234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":2623846234},{"type":"v1","addr":"172.21.15.194:6837","nonce":2623846234}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/491309336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":491309336},{"type":"v1","addr":"172.21.15.123:6837","nonce":491309336}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[]} 2024-01-24T14:24:52.494 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.494+0000 7fe52e51e700 1 -- 172.21.15.19:0/1263797109 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fe5100a05f0 msgr2=0x7fe5100a2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:52.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.494+0000 7fe52e51e700 1 --2- 172.21.15.19:0/1263797109 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fe5100a05f0 0x7fe5100a2ab0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe518007c60 tx=0x7fe5180073d0 comp rx=0 tx=0).stop 2024-01-24T14:24:52.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.494+0000 7fe52e51e700 1 -- 172.21.15.19:0/1263797109 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe5281040d0 msgr2=0x7fe528196fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:52.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.494+0000 7fe52e51e700 1 --2- 172.21.15.19:0/1263797109 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe5281040d0 0x7fe528196fe0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe51c00d950 tx=0x7fe51c00dc60 comp rx=0 tx=0).stop 2024-01-24T14:24:52.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.494+0000 7fe52e51e700 1 -- 172.21.15.19:0/1263797109 shutdown_connections 2024-01-24T14:24:52.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.494+0000 7fe52e51e700 1 --2- 172.21.15.19:0/1263797109 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe528106a00 0x7fe52819e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:52.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.494+0000 7fe52e51e700 1 --2- 172.21.15.19:0/1263797109 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe5281017a0 0x7fe528196aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:52.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.494+0000 7fe52e51e700 1 --2- 172.21.15.19:0/1263797109 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fe5100a05f0 0x7fe5100a2ab0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:52.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.494+0000 7fe52e51e700 1 --2- 172.21.15.19:0/1263797109 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe5281040d0 0x7fe528196fe0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:52.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.494+0000 7fe52e51e700 1 -- 172.21.15.19:0/1263797109 >> 172.21.15.19:0/1263797109 conn(0x7fe5280fb170 msgr2=0x7fe528102410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:24:52.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.494+0000 7fe52e51e700 1 -- 172.21.15.19:0/1263797109 shutdown_connections 2024-01-24T14:24:52.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:52.494+0000 7fe52e51e700 1 -- 172.21.15.19:0/1263797109 wait complete. 2024-01-24T14:24:52.496 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 9 2024-01-24T14:24:53.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:53 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:53.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:53 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:53.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:53 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:53.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:53 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T14:24:53.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:53 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T14:24:53.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:53 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:53.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:53 smithi019 ceph-mon[40663]: Deploying daemon mds.g on smithi019 2024-01-24T14:24:53.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:53 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/1263797109' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T14:24:53.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:53 smithi019 ceph-mon[40663]: mds.? [v2:172.21.15.194:6838/810249223,v1:172.21.15.194:6839/810249223] up:boot 2024-01-24T14:24:53.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:53 smithi019 ceph-mon[40663]: fsmap 9 up:standby 2024-01-24T14:24:53.237 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:53 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-01-24T14:24:53.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:53 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:53.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:53 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:53.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:53 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:53.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:53 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T14:24:53.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:53 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T14:24:53.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:53 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:53.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:53 smithi194 ceph-mon[45450]: Deploying daemon mds.g on smithi019 2024-01-24T14:24:53.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:53 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/1263797109' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T14:24:53.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:53 smithi194 ceph-mon[45450]: mds.? [v2:172.21.15.194:6838/810249223,v1:172.21.15.194:6839/810249223] up:boot 2024-01-24T14:24:53.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:53 smithi194 ceph-mon[45450]: fsmap 9 up:standby 2024-01-24T14:24:53.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:53 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-01-24T14:24:53.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:53 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:53.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:53 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:53.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:53 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:53.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:53 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T14:24:53.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:53 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T14:24:53.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:53 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:53.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:53 smithi123 ceph-mon[44966]: Deploying daemon mds.g on smithi019 2024-01-24T14:24:53.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:53 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/1263797109' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T14:24:53.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:53 smithi123 ceph-mon[44966]: mds.? [v2:172.21.15.194:6838/810249223,v1:172.21.15.194:6839/810249223] up:boot 2024-01-24T14:24:53.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:53 smithi123 ceph-mon[44966]: fsmap 9 up:standby 2024-01-24T14:24:53.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:53 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-01-24T14:24:53.734 INFO:journalctl@ceph.mds.g.smithi019.stdout:Jan 24 14:24:53 smithi019 systemd[1]: Starting Ceph mds.g for 654b0c8e-bac3-11ee-95b1-87774f69a715... 2024-01-24T14:24:54.236 INFO:journalctl@ceph.mds.g.smithi019.stdout:Jan 24 14:24:53 smithi019 ceph-mds[93107]: -- 172.21.15.19:0/1638293720 <== mon.0 v2:172.21.15.19:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556eff4cb1e0 con 0x556eff4a0800 2024-01-24T14:24:54.236 INFO:journalctl@ceph.mds.g.smithi019.stdout:Jan 24 14:24:53 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-g[93084]: starting mds.g at 2024-01-24T14:24:54.236 INFO:journalctl@ceph.mds.g.smithi019.stdout:Jan 24 14:24:53 smithi019 bash[93070]: 3d229e2df07f102dda770919168ec00a678779726fc74f43307f8e5fe73d530a 2024-01-24T14:24:54.236 INFO:journalctl@ceph.mds.g.smithi019.stdout:Jan 24 14:24:53 smithi019 systemd[1]: Started Ceph mds.g for 654b0c8e-bac3-11ee-95b1-87774f69a715. 2024-01-24T14:24:54.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:54 smithi019 ceph-mon[40663]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:24:54.237 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:54 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:54.237 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:54 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:54.237 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:54 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:54.237 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:54 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T14:24:54.237 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:54 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T14:24:54.237 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:54 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:54.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:54 smithi194 ceph-mon[45450]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:24:54.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:54 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:54.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:54 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:54.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:54 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:54.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:54 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T14:24:54.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:54 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T14:24:54.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:54 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:54.509 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T14:24:54.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:54 smithi123 ceph-mon[44966]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:24:54.549 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:54 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:54.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:54 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:54.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:54 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:54.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:54 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T14:24:54.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:54 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T14:24:54.550 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:54 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:54.664 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.664+0000 7f28c71cc700 1 -- 172.21.15.19:0/3871258362 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f28c0100e40 msgr2=0x7f28c01012a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:54.665 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.664+0000 7f28c71cc700 1 --2- 172.21.15.19:0/3871258362 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f28c0100e40 0x7f28c01012a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f28b40099c0 tx=0x7f28b4009cd0 comp rx=0 tx=0).stop 2024-01-24T14:24:54.665 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.664+0000 7f28c71cc700 1 -- 172.21.15.19:0/3871258362 shutdown_connections 2024-01-24T14:24:54.665 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.664+0000 7f28c71cc700 1 --2- 172.21.15.19:0/3871258362 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f28c01017e0 0x7f28c0108090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:54.665 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.664+0000 7f28c71cc700 1 --2- 172.21.15.19:0/3871258362 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28c00ffc50 0x7f28c0100070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:54.665 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.664+0000 7f28c71cc700 1 --2- 172.21.15.19:0/3871258362 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f28c0100e40 0x7f28c01012a0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:54.665 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.664+0000 7f28c71cc700 1 -- 172.21.15.19:0/3871258362 >> 172.21.15.19:0/3871258362 conn(0x7f28c00fb1c0 msgr2=0x7f28c00fd600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:24:54.665 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.664+0000 7f28c71cc700 1 -- 172.21.15.19:0/3871258362 shutdown_connections 2024-01-24T14:24:54.665 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.664+0000 7f28c71cc700 1 -- 172.21.15.19:0/3871258362 wait complete. 2024-01-24T14:24:54.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.665+0000 7f28c71cc700 1 Processor -- start 2024-01-24T14:24:54.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.665+0000 7f28c71cc700 1 -- start start 2024-01-24T14:24:54.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.665+0000 7f28c71cc700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f28c00ffc50 0x7f28c0103be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:54.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.665+0000 7f28c71cc700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f28c0100e40 0x7f28c0102220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:54.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.665+0000 7f28c71cc700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28c01017e0 0x7f28c0102760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:54.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.665+0000 7f28c71cc700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c0102f00 con 0x7f28c00ffc50 2024-01-24T14:24:54.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.665+0000 7f28c71cc700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c0103070 con 0x7f28c01017e0 2024-01-24T14:24:54.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.665+0000 7f28c71cc700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c01a0a90 con 0x7f28c0100e40 2024-01-24T14:24:54.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.665+0000 7f28c4f68700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f28c00ffc50 0x7f28c0103be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:54.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.665+0000 7f28c4f68700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f28c00ffc50 0x7f28c0103be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:54174/0 (socket says 172.21.15.19:54174) 2024-01-24T14:24:54.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.665+0000 7f28c4f68700 1 -- 172.21.15.19:0/3023691202 learned_addr learned my addr 172.21.15.19:0/3023691202 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:24:54.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.665+0000 7f28c5769700 1 --2- 172.21.15.19:0/3023691202 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28c01017e0 0x7f28c0102760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:54.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.665+0000 7f28bffff700 1 --2- 172.21.15.19:0/3023691202 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f28c0100e40 0x7f28c0102220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:54.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.666+0000 7f28c4f68700 1 -- 172.21.15.19:0/3023691202 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f28c0100e40 msgr2=0x7f28c0102220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:54.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.666+0000 7f28c4f68700 1 --2- 172.21.15.19:0/3023691202 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f28c0100e40 0x7f28c0102220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:54.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.666+0000 7f28c4f68700 1 -- 172.21.15.19:0/3023691202 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28c01017e0 msgr2=0x7f28c0102760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:54.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.666+0000 7f28c4f68700 1 --2- 172.21.15.19:0/3023691202 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28c01017e0 0x7f28c0102760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:54.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.666+0000 7f28c4f68700 1 -- 172.21.15.19:0/3023691202 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28b40095f0 con 0x7f28c00ffc50 2024-01-24T14:24:54.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.666+0000 7f28c4f68700 1 --2- 172.21.15.19:0/3023691202 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f28c00ffc50 0x7f28c0103be0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f28b000e970 tx=0x7f28b000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:24:54.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.666+0000 7f28bdffb700 1 -- 172.21.15.19:0/3023691202 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b0009790 con 0x7f28c00ffc50 2024-01-24T14:24:54.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.666+0000 7f28bdffb700 1 -- 172.21.15.19:0/3023691202 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f28b000c7e0 con 0x7f28c00ffc50 2024-01-24T14:24:54.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.666+0000 7f28c71cc700 1 -- 172.21.15.19:0/3023691202 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28c01a0d70 con 0x7f28c00ffc50 2024-01-24T14:24:54.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.666+0000 7f28bdffb700 1 -- 172.21.15.19:0/3023691202 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b0003c70 con 0x7f28c00ffc50 2024-01-24T14:24:54.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.666+0000 7f28c71cc700 1 -- 172.21.15.19:0/3023691202 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28c01a11d0 con 0x7f28c00ffc50 2024-01-24T14:24:54.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.667+0000 7f28c71cc700 1 -- 172.21.15.19:0/3023691202 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28c010b460 con 0x7f28c00ffc50 2024-01-24T14:24:54.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.668+0000 7f28bdffb700 1 -- 172.21.15.19:0/3023691202 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f28b000cc50 con 0x7f28c00ffc50 2024-01-24T14:24:54.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.669+0000 7f28bdffb700 1 --2- 172.21.15.19:0/3023691202 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f28a80a0570 0x7f28a80a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:54.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.669+0000 7f28bdffb700 1 -- 172.21.15.19:0/3023691202 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f28b0014070 con 0x7f28c00ffc50 2024-01-24T14:24:54.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.669+0000 7f28bffff700 1 --2- 172.21.15.19:0/3023691202 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f28a80a0570 0x7f28a80a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:54.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.669+0000 7f28bffff700 1 --2- 172.21.15.19:0/3023691202 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f28a80a0570 0x7f28a80a2a30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f28b400b580 tx=0x7f28b4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:24:54.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.673+0000 7f28bdffb700 1 -- 172.21.15.19:0/3023691202 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f28b00956e0 con 0x7f28c00ffc50 2024-01-24T14:24:54.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.951+0000 7f28c71cc700 1 -- 172.21.15.19:0/3023691202 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28c010b620 con 0x7f28c00ffc50 2024-01-24T14:24:54.952 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.952+0000 7f28bdffb700 1 -- 172.21.15.19:0/3023691202 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8085 (secure 0 0 0) 0x7f28b00ace60 con 0x7f28c00ffc50 2024-01-24T14:24:54.952 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:24:54.953 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6835/1060583512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6834","nonce":1060583512},{"type":"v1","addr":"172.21.15.19:6835","nonce":1060583512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6837/2302964875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6836","nonce":2302964875},{"type":"v1","addr":"172.21.15.19:6837","nonce":2302964875}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14607,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6839/3376327575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6838","nonce":3376327575},{"type":"v1","addr":"172.21.15.19:6839","nonce":3376327575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2647625404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2647625404},{"type":"v1","addr":"172.21.15.194:6833","nonce":2647625404}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/3396650565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3396650565},{"type":"v1","addr":"172.21.15.123:6833","nonce":3396650565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24485,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2631676083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2631676083},{"type":"v1","addr":"172.21.15.123:6835","nonce":2631676083}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6835/456696823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":456696823},{"type":"v1","addr":"172.21.15.194:6835","nonce":456696823}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6837/2623846234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":2623846234},{"type":"v1","addr":"172.21.15.194:6837","nonce":2623846234}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/491309336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":491309336},{"type":"v1","addr":"172.21.15.123:6837","nonce":491309336}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24514,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6839/810249223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":810249223},{"type":"v1","addr":"172.21.15.194:6839","nonce":810249223}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2024-01-24T14:24:54.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.955+0000 7f28c71cc700 1 -- 172.21.15.19:0/3023691202 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f28a80a0570 msgr2=0x7f28a80a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:54.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.955+0000 7f28c71cc700 1 --2- 172.21.15.19:0/3023691202 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f28a80a0570 0x7f28a80a2a30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f28b400b580 tx=0x7f28b4005fb0 comp rx=0 tx=0).stop 2024-01-24T14:24:54.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.955+0000 7f28c71cc700 1 -- 172.21.15.19:0/3023691202 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f28c00ffc50 msgr2=0x7f28c0103be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:54.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.955+0000 7f28c71cc700 1 --2- 172.21.15.19:0/3023691202 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f28c00ffc50 0x7f28c0103be0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f28b000e970 tx=0x7f28b000ec80 comp rx=0 tx=0).stop 2024-01-24T14:24:54.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.955+0000 7f28c71cc700 1 -- 172.21.15.19:0/3023691202 shutdown_connections 2024-01-24T14:24:54.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.955+0000 7f28c71cc700 1 --2- 172.21.15.19:0/3023691202 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f28a80a0570 0x7f28a80a2a30 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:54.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.955+0000 7f28c71cc700 1 --2- 172.21.15.19:0/3023691202 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f28c00ffc50 0x7f28c0103be0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:54.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.955+0000 7f28c71cc700 1 --2- 172.21.15.19:0/3023691202 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28c01017e0 0x7f28c0102760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:54.956 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.955+0000 7f28c71cc700 1 --2- 172.21.15.19:0/3023691202 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f28c0100e40 0x7f28c0102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:54.956 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.955+0000 7f28c71cc700 1 -- 172.21.15.19:0/3023691202 >> 172.21.15.19:0/3023691202 conn(0x7f28c00fb1c0 msgr2=0x7f28c0069140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:24:54.956 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.955+0000 7f28c71cc700 1 -- 172.21.15.19:0/3023691202 shutdown_connections 2024-01-24T14:24:54.956 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:54.955+0000 7f28c71cc700 1 -- 172.21.15.19:0/3023691202 wait complete. 2024-01-24T14:24:54.956 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 11 2024-01-24T14:24:55.235 INFO:journalctl@ceph.mds.a.smithi019.stdout:Jan 24 14:24:55 smithi019 systemd[1]: Starting Ceph mds.a for 654b0c8e-bac3-11ee-95b1-87774f69a715... 2024-01-24T14:24:55.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:55 smithi019 ceph-mon[40663]: Deploying daemon mds.a on smithi019 2024-01-24T14:24:55.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:55 smithi019 ceph-mon[40663]: mds.? [v2:172.21.15.19:6838/3376327575,v1:172.21.15.19:6839/3376327575] up:boot 2024-01-24T14:24:55.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:55 smithi019 ceph-mon[40663]: fsmap 10 up:standby 2024-01-24T14:24:55.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:55 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-01-24T14:24:55.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:55 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/3023691202' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T14:24:55.736 INFO:journalctl@ceph.mds.a.smithi019.stdout:Jan 24 14:24:55 smithi019 bash[93456]: df0acf93d164d14fa059740b217d0dcf95a1d28e8fb35504f1e9b24953a2b197 2024-01-24T14:24:55.736 INFO:journalctl@ceph.mds.a.smithi019.stdout:Jan 24 14:24:55 smithi019 ceph-mds[93494]: -- 172.21.15.19:0/1059940163 <== mon.0 v2:172.21.15.19:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55da79db91e0 con 0x55da79d8e800 2024-01-24T14:24:55.736 INFO:journalctl@ceph.mds.a.smithi019.stdout:Jan 24 14:24:55 smithi019 systemd[1]: Started Ceph mds.a for 654b0c8e-bac3-11ee-95b1-87774f69a715. 2024-01-24T14:24:55.737 INFO:journalctl@ceph.mds.a.smithi019.stdout:Jan 24 14:24:55 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-a[93471]: starting mds.a at 2024-01-24T14:24:55.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:55 smithi123 ceph-mon[44966]: Deploying daemon mds.a on smithi019 2024-01-24T14:24:55.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:55 smithi123 ceph-mon[44966]: mds.? [v2:172.21.15.19:6838/3376327575,v1:172.21.15.19:6839/3376327575] up:boot 2024-01-24T14:24:55.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:55 smithi123 ceph-mon[44966]: fsmap 10 up:standby 2024-01-24T14:24:55.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:55 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-01-24T14:24:55.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:55 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/3023691202' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T14:24:55.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:55 smithi194 ceph-mon[45450]: Deploying daemon mds.a on smithi019 2024-01-24T14:24:55.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:55 smithi194 ceph-mon[45450]: mds.? [v2:172.21.15.19:6838/3376327575,v1:172.21.15.19:6839/3376327575] up:boot 2024-01-24T14:24:55.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:55 smithi194 ceph-mon[45450]: fsmap 10 up:standby 2024-01-24T14:24:55.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:55 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-01-24T14:24:55.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:55 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/3023691202' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T14:24:56.714 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:56 smithi123 ceph-mon[44966]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:24:56.714 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:56 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:56.714 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:56 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:56.714 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:56 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:56.714 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:56 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T14:24:56.714 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:56 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T14:24:56.714 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:56 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:56.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:56 smithi019 ceph-mon[40663]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:24:56.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:56 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:56.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:56 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:56.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:56 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:56.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:56 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T14:24:56.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:56 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T14:24:56.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:56 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:56.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:56 smithi194 ceph-mon[45450]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:24:56.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:56 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:56.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:56 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:56.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:56 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:56.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:56 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-01-24T14:24:56.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:56 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-01-24T14:24:56.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:56 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:24:56.970 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T14:24:57.122 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.122+0000 7f64d248d700 1 -- 172.21.15.19:0/330722850 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f64cc1040d0 msgr2=0x7f64cc1064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:57.122 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.122+0000 7f64d248d700 1 --2- 172.21.15.19:0/330722850 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f64cc1040d0 0x7f64cc1064c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f64c0009990 tx=0x7f64c0009ca0 comp rx=0 tx=0).stop 2024-01-24T14:24:57.122 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.122+0000 7f64d248d700 1 -- 172.21.15.19:0/330722850 shutdown_connections 2024-01-24T14:24:57.122 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.122+0000 7f64d248d700 1 --2- 172.21.15.19:0/330722850 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64cc106a00 0x7f64cc108e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:57.122 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.122+0000 7f64d248d700 1 --2- 172.21.15.19:0/330722850 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f64cc1017a0 0x7f64cc103b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:57.122 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.122+0000 7f64d248d700 1 --2- 172.21.15.19:0/330722850 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f64cc1040d0 0x7f64cc1064c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:57.123 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.122+0000 7f64d248d700 1 -- 172.21.15.19:0/330722850 >> 172.21.15.19:0/330722850 conn(0x7f64cc0fb170 msgr2=0x7f64cc0fd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:24:57.123 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.122+0000 7f64d248d700 1 -- 172.21.15.19:0/330722850 shutdown_connections 2024-01-24T14:24:57.123 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.122+0000 7f64d248d700 1 -- 172.21.15.19:0/330722850 wait complete. 2024-01-24T14:24:57.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.122+0000 7f64d248d700 1 Processor -- start 2024-01-24T14:24:57.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.123+0000 7f64d248d700 1 -- start start 2024-01-24T14:24:57.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.123+0000 7f64d248d700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f64cc1017a0 0x7f64cc1969b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:57.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.123+0000 7f64d248d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64cc1040d0 0x7f64cc196ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:57.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.123+0000 7f64d248d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f64cc106a00 0x7f64cc19df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:57.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.123+0000 7f64d248d700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64cc19e530 con 0x7f64cc1017a0 2024-01-24T14:24:57.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.123+0000 7f64d248d700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64cc19e670 con 0x7f64cc1040d0 2024-01-24T14:24:57.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.123+0000 7f64d248d700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64cc19e800 con 0x7f64cc106a00 2024-01-24T14:24:57.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.123+0000 7f64cbfff700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f64cc1017a0 0x7f64cc1969b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:57.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.123+0000 7f64cbfff700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f64cc1017a0 0x7f64cc1969b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:54212/0 (socket says 172.21.15.19:54212) 2024-01-24T14:24:57.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.123+0000 7f64cbfff700 1 -- 172.21.15.19:0/3899630291 learned_addr learned my addr 172.21.15.19:0/3899630291 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:24:57.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.123+0000 7f64cb7fe700 1 --2- 172.21.15.19:0/3899630291 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64cc1040d0 0x7f64cc196ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:57.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.123+0000 7f64d0a2a700 1 --2- 172.21.15.19:0/3899630291 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f64cc106a00 0x7f64cc19df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:57.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.123+0000 7f64cbfff700 1 -- 172.21.15.19:0/3899630291 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f64cc106a00 msgr2=0x7f64cc19df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:57.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.123+0000 7f64cbfff700 1 --2- 172.21.15.19:0/3899630291 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f64cc106a00 0x7f64cc19df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:57.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.123+0000 7f64cbfff700 1 -- 172.21.15.19:0/3899630291 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64cc1040d0 msgr2=0x7f64cc196ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:57.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.123+0000 7f64cbfff700 1 --2- 172.21.15.19:0/3899630291 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64cc1040d0 0x7f64cc196ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:57.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.123+0000 7f64cbfff700 1 -- 172.21.15.19:0/3899630291 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c00095f0 con 0x7f64cc1017a0 2024-01-24T14:24:57.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.123+0000 7f64cb7fe700 1 --2- 172.21.15.19:0/3899630291 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64cc1040d0 0x7f64cc196ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T14:24:57.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.124+0000 7f64cbfff700 1 --2- 172.21.15.19:0/3899630291 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f64cc1017a0 0x7f64cc1969b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f64bc00e9a0 tx=0x7f64bc00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:24:57.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.124+0000 7f64c97fa700 1 -- 172.21.15.19:0/3899630291 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64bc009790 con 0x7f64cc1017a0 2024-01-24T14:24:57.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.124+0000 7f64d248d700 1 -- 172.21.15.19:0/3899630291 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64cc19eae0 con 0x7f64cc1017a0 2024-01-24T14:24:57.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.124+0000 7f64c97fa700 1 -- 172.21.15.19:0/3899630291 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f64bc00cbd0 con 0x7f64cc1017a0 2024-01-24T14:24:57.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.124+0000 7f64c97fa700 1 -- 172.21.15.19:0/3899630291 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64bc018680 con 0x7f64cc1017a0 2024-01-24T14:24:57.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.124+0000 7f64d248d700 1 -- 172.21.15.19:0/3899630291 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64cc19eff0 con 0x7f64cc1017a0 2024-01-24T14:24:57.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.125+0000 7f64d248d700 1 -- 172.21.15.19:0/3899630291 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64cc10c510 con 0x7f64cc1017a0 2024-01-24T14:24:57.128 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.126+0000 7f64c97fa700 1 -- 172.21.15.19:0/3899630291 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f64bc00cd40 con 0x7f64cc1017a0 2024-01-24T14:24:57.128 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.127+0000 7f64c97fa700 1 --2- 172.21.15.19:0/3899630291 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f64b40a0570 0x7f64b40a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:57.128 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.127+0000 7f64cb7fe700 1 --2- 172.21.15.19:0/3899630291 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f64b40a0570 0x7f64b40a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:57.128 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.127+0000 7f64c97fa700 1 -- 172.21.15.19:0/3899630291 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f64bc014070 con 0x7f64cc1017a0 2024-01-24T14:24:57.128 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.127+0000 7f64cb7fe700 1 --2- 172.21.15.19:0/3899630291 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f64b40a0570 0x7f64b40a2a30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f64c000b2e0 tx=0x7f64c0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:24:57.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.130+0000 7f64c97fa700 1 -- 172.21.15.19:0/3899630291 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f64bc095d20 con 0x7f64cc1017a0 2024-01-24T14:24:57.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.393+0000 7f64d248d700 1 -- 172.21.15.19:0/3899630291 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64cc10c700 con 0x7f64cc1017a0 2024-01-24T14:24:57.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.393+0000 7f64c97fa700 1 -- 172.21.15.19:0/3899630291 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+8843 (secure 0 0 0) 0x7f64bc099340 con 0x7f64cc1017a0 2024-01-24T14:24:57.394 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:24:57.394 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6835/1060583512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6834","nonce":1060583512},{"type":"v1","addr":"172.21.15.19:6835","nonce":1060583512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6837/2302964875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6836","nonce":2302964875},{"type":"v1","addr":"172.21.15.19:6837","nonce":2302964875}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14607,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6839/3376327575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6838","nonce":3376327575},{"type":"v1","addr":"172.21.15.19:6839","nonce":3376327575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14619,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6841/1702088084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6840","nonce":1702088084},{"type":"v1","addr":"172.21.15.19:6841","nonce":1702088084}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2647625404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2647625404},{"type":"v1","addr":"172.21.15.194:6833","nonce":2647625404}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/3396650565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3396650565},{"type":"v1","addr":"172.21.15.123:6833","nonce":3396650565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24485,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2631676083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2631676083},{"type":"v1","addr":"172.21.15.123:6835","nonce":2631676083}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6835/456696823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":456696823},{"type":"v1","addr":"172.21.15.194:6835","nonce":456696823}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6837/2623846234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":2623846234},{"type":"v1","addr":"172.21.15.194:6837","nonce":2623846234}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/491309336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":491309336},{"type":"v1","addr":"172.21.15.123:6837","nonce":491309336}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24514,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6839/810249223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":810249223},{"type":"v1","addr":"172.21.15.194:6839","nonce":810249223}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2024-01-24T14:24:57.396 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.396+0000 7f64d248d700 1 -- 172.21.15.19:0/3899630291 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f64b40a0570 msgr2=0x7f64b40a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:57.396 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.396+0000 7f64d248d700 1 --2- 172.21.15.19:0/3899630291 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f64b40a0570 0x7f64b40a2a30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f64c000b2e0 tx=0x7f64c0005fb0 comp rx=0 tx=0).stop 2024-01-24T14:24:57.396 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.396+0000 7f64d248d700 1 -- 172.21.15.19:0/3899630291 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f64cc1017a0 msgr2=0x7f64cc1969b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:57.396 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.396+0000 7f64d248d700 1 --2- 172.21.15.19:0/3899630291 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f64cc1017a0 0x7f64cc1969b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f64bc00e9a0 tx=0x7f64bc00ed60 comp rx=0 tx=0).stop 2024-01-24T14:24:57.396 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.396+0000 7f64d248d700 1 -- 172.21.15.19:0/3899630291 shutdown_connections 2024-01-24T14:24:57.396 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.396+0000 7f64d248d700 1 --2- 172.21.15.19:0/3899630291 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f64b40a0570 0x7f64b40a2a30 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:57.396 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.396+0000 7f64d248d700 1 --2- 172.21.15.19:0/3899630291 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f64cc1017a0 0x7f64cc1969b0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:57.396 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.396+0000 7f64d248d700 1 --2- 172.21.15.19:0/3899630291 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f64cc106a00 0x7f64cc19df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:57.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.396+0000 7f64d248d700 1 --2- 172.21.15.19:0/3899630291 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64cc1040d0 0x7f64cc196ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:57.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.396+0000 7f64d248d700 1 -- 172.21.15.19:0/3899630291 >> 172.21.15.19:0/3899630291 conn(0x7f64cc0fb170 msgr2=0x7f64cc102c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:24:57.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.396+0000 7f64d248d700 1 -- 172.21.15.19:0/3899630291 shutdown_connections 2024-01-24T14:24:57.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:57.396+0000 7f64d248d700 1 -- 172.21.15.19:0/3899630291 wait complete. 2024-01-24T14:24:57.397 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 12 2024-01-24T14:24:57.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:57 smithi019 ceph-mon[40663]: Deploying daemon mds.b on smithi123 2024-01-24T14:24:57.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:57 smithi019 ceph-mon[40663]: mds.? [v2:172.21.15.19:6840/1702088084,v1:172.21.15.19:6841/1702088084] up:boot 2024-01-24T14:24:57.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:57 smithi019 ceph-mon[40663]: fsmap 11 up:standby 2024-01-24T14:24:57.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:57 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-01-24T14:24:57.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:57 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/3899630291' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T14:24:57.799 INFO:journalctl@ceph.mds.b.smithi123.stdout:Jan 24 14:24:57 smithi123 bash[81345]: 35e5e20ecc506e4b0e366d6f275430cfe5de19a9d13e40505e9959734474ba34 2024-01-24T14:24:57.800 INFO:journalctl@ceph.mds.b.smithi123.stdout:Jan 24 14:24:57 smithi123 systemd[1]: Started Ceph mds.b for 654b0c8e-bac3-11ee-95b1-87774f69a715. 2024-01-24T14:24:57.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:57 smithi123 ceph-mon[44966]: Deploying daemon mds.b on smithi123 2024-01-24T14:24:57.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:57 smithi123 ceph-mon[44966]: mds.? [v2:172.21.15.19:6840/1702088084,v1:172.21.15.19:6841/1702088084] up:boot 2024-01-24T14:24:57.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:57 smithi123 ceph-mon[44966]: fsmap 11 up:standby 2024-01-24T14:24:57.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:57 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-01-24T14:24:57.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:57 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/3899630291' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T14:24:57.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:57 smithi194 ceph-mon[45450]: Deploying daemon mds.b on smithi123 2024-01-24T14:24:57.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:57 smithi194 ceph-mon[45450]: mds.? [v2:172.21.15.19:6840/1702088084,v1:172.21.15.19:6841/1702088084] up:boot 2024-01-24T14:24:57.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:57 smithi194 ceph-mon[45450]: fsmap 11 up:standby 2024-01-24T14:24:57.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:57 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-01-24T14:24:57.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:57 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/3899630291' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T14:24:58.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:58 smithi019 ceph-mon[40663]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:24:58.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:58 smithi019 ceph-mon[40663]: mds.? [v2:172.21.15.123:6838/4289194720,v1:172.21.15.123:6839/4289194720] up:boot 2024-01-24T14:24:58.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:58 smithi019 ceph-mon[40663]: fsmap 12 up:standby 2024-01-24T14:24:58.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:58 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-01-24T14:24:58.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:58 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:58.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:58 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:58.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:58 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:58.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:58 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:58.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:58 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:58.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:58 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:24:58.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:58 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:58.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:58 smithi123 ceph-mon[44966]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:24:58.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:58 smithi123 ceph-mon[44966]: mds.? [v2:172.21.15.123:6838/4289194720,v1:172.21.15.123:6839/4289194720] up:boot 2024-01-24T14:24:58.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:58 smithi123 ceph-mon[44966]: fsmap 12 up:standby 2024-01-24T14:24:58.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:58 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-01-24T14:24:58.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:58 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:58.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:58 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:58.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:58 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:58.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:58 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:58.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:58 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:58.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:58 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:24:58.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:58 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:58.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:58 smithi194 ceph-mon[45450]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:24:58.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:58 smithi194 ceph-mon[45450]: mds.? [v2:172.21.15.123:6838/4289194720,v1:172.21.15.123:6839/4289194720] up:boot 2024-01-24T14:24:58.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:58 smithi194 ceph-mon[45450]: fsmap 12 up:standby 2024-01-24T14:24:58.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:58 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-01-24T14:24:58.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:58 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:58.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:58 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:58.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:58 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:58.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:58 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:58.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:58 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:58.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:58 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:24:58.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:58 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:24:59.411 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T14:24:59.564 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.564+0000 7fca88342700 1 -- 172.21.15.19:0/278130252 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fca80106a00 msgr2=0x7fca80108e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:59.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.564+0000 7fca88342700 1 --2- 172.21.15.19:0/278130252 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fca80106a00 0x7fca80108e70 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fca7800b0a0 tx=0x7fca7800b3b0 comp rx=0 tx=0).stop 2024-01-24T14:24:59.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.564+0000 7fca88342700 1 -- 172.21.15.19:0/278130252 shutdown_connections 2024-01-24T14:24:59.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.564+0000 7fca88342700 1 --2- 172.21.15.19:0/278130252 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fca80106a00 0x7fca80108e70 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:59.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.564+0000 7fca88342700 1 --2- 172.21.15.19:0/278130252 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fca801017a0 0x7fca80103b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:59.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.564+0000 7fca88342700 1 --2- 172.21.15.19:0/278130252 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca801040d0 0x7fca801064c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:59.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.564+0000 7fca88342700 1 -- 172.21.15.19:0/278130252 >> 172.21.15.19:0/278130252 conn(0x7fca800fb170 msgr2=0x7fca800fd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:24:59.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.564+0000 7fca88342700 1 -- 172.21.15.19:0/278130252 shutdown_connections 2024-01-24T14:24:59.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.564+0000 7fca88342700 1 -- 172.21.15.19:0/278130252 wait complete. 2024-01-24T14:24:59.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.565+0000 7fca88342700 1 Processor -- start 2024-01-24T14:24:59.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.565+0000 7fca88342700 1 -- start start 2024-01-24T14:24:59.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.565+0000 7fca88342700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fca801017a0 0x7fca80101110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:59.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.565+0000 7fca88342700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca801040d0 0x7fca800ff750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:59.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.565+0000 7fca88342700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fca80106a00 0x7fca800ffc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:59.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.565+0000 7fca88342700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca80100430 con 0x7fca801017a0 2024-01-24T14:24:59.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.565+0000 7fca88342700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca801005a0 con 0x7fca801040d0 2024-01-24T14:24:59.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.565+0000 7fca88342700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca801a0b00 con 0x7fca80106a00 2024-01-24T14:24:59.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.566+0000 7fca860de700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fca801017a0 0x7fca80101110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:59.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.566+0000 7fca860de700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fca801017a0 0x7fca80101110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:54228/0 (socket says 172.21.15.19:54228) 2024-01-24T14:24:59.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.566+0000 7fca860de700 1 -- 172.21.15.19:0/731359826 learned_addr learned my addr 172.21.15.19:0/731359826 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:24:59.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.566+0000 7fca858dd700 1 --2- 172.21.15.19:0/731359826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca801040d0 0x7fca800ff750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:59.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.566+0000 7fca868df700 1 --2- 172.21.15.19:0/731359826 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fca80106a00 0x7fca800ffc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:59.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.566+0000 7fca858dd700 1 -- 172.21.15.19:0/731359826 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fca80106a00 msgr2=0x7fca800ffc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:59.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.566+0000 7fca858dd700 1 --2- 172.21.15.19:0/731359826 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fca80106a00 0x7fca800ffc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:59.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.566+0000 7fca858dd700 1 -- 172.21.15.19:0/731359826 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fca801017a0 msgr2=0x7fca80101110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:59.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.566+0000 7fca858dd700 1 --2- 172.21.15.19:0/731359826 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fca801017a0 0x7fca80101110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:59.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.566+0000 7fca858dd700 1 -- 172.21.15.19:0/731359826 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca70009520 con 0x7fca801040d0 2024-01-24T14:24:59.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.566+0000 7fca860de700 1 --2- 172.21.15.19:0/731359826 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fca801017a0 0x7fca80101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T14:24:59.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.566+0000 7fca858dd700 1 --2- 172.21.15.19:0/731359826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca801040d0 0x7fca800ff750 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fca7000e9f0 tx=0x7fca7000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:24:59.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.566+0000 7fca777fe700 1 -- 172.21.15.19:0/731359826 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca7000cb70 con 0x7fca801040d0 2024-01-24T14:24:59.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.566+0000 7fca777fe700 1 -- 172.21.15.19:0/731359826 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fca7000ee90 con 0x7fca801040d0 2024-01-24T14:24:59.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.566+0000 7fca88342700 1 -- 172.21.15.19:0/731359826 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca78009d00 con 0x7fca801040d0 2024-01-24T14:24:59.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.566+0000 7fca777fe700 1 -- 172.21.15.19:0/731359826 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca70003cf0 con 0x7fca801040d0 2024-01-24T14:24:59.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.566+0000 7fca88342700 1 -- 172.21.15.19:0/731359826 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca801a11b0 con 0x7fca801040d0 2024-01-24T14:24:59.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.567+0000 7fca88342700 1 -- 172.21.15.19:0/731359826 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca800fccf0 con 0x7fca801040d0 2024-01-24T14:24:59.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.569+0000 7fca777fe700 1 -- 172.21.15.19:0/731359826 <== mon.2 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7fca7000ccd0 con 0x7fca801040d0 2024-01-24T14:24:59.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.569+0000 7fca777fe700 1 --2- 172.21.15.19:0/731359826 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fca6c0a05f0 0x7fca6c0a2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:24:59.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.569+0000 7fca860de700 1 --2- 172.21.15.19:0/731359826 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fca6c0a05f0 0x7fca6c0a2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:24:59.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.569+0000 7fca777fe700 1 -- 172.21.15.19:0/731359826 <== mon.2 v2:172.21.15.123:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fca70014070 con 0x7fca801040d0 2024-01-24T14:24:59.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.570+0000 7fca860de700 1 --2- 172.21.15.19:0/731359826 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fca6c0a05f0 0x7fca6c0a2ab0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fca7c011550 tx=0x7fca7c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:24:59.572 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.572+0000 7fca777fe700 1 -- 172.21.15.19:0/731359826 <== mon.2 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fca70095830 con 0x7fca801040d0 2024-01-24T14:24:59.885 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.883+0000 7fca88342700 1 -- 172.21.15.19:0/731359826 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca8002cc70 con 0x7fca801040d0 2024-01-24T14:24:59.885 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.884+0000 7fca777fe700 1 -- 172.21.15.19:0/731359826 <== mon.2 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+9604 (secure 0 0 0) 0x7fca70098e50 con 0x7fca801040d0 2024-01-24T14:24:59.885 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:24:59.885 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6835/1060583512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6834","nonce":1060583512},{"type":"v1","addr":"172.21.15.19:6835","nonce":1060583512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6837/2302964875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6836","nonce":2302964875},{"type":"v1","addr":"172.21.15.19:6837","nonce":2302964875}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14607,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6839/3376327575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6838","nonce":3376327575},{"type":"v1","addr":"172.21.15.19:6839","nonce":3376327575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14619,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6841/1702088084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6840","nonce":1702088084},{"type":"v1","addr":"172.21.15.19:6841","nonce":1702088084}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2647625404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2647625404},{"type":"v1","addr":"172.21.15.194:6833","nonce":2647625404}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/3396650565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3396650565},{"type":"v1","addr":"172.21.15.123:6833","nonce":3396650565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24485,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2631676083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2631676083},{"type":"v1","addr":"172.21.15.123:6835","nonce":2631676083}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6835/456696823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":456696823},{"type":"v1","addr":"172.21.15.194:6835","nonce":456696823}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6837/2623846234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":2623846234},{"type":"v1","addr":"172.21.15.194:6837","nonce":2623846234}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/491309336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":491309336},{"type":"v1","addr":"172.21.15.123:6837","nonce":491309336}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24514,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6839/810249223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":810249223},{"type":"v1","addr":"172.21.15.194:6839","nonce":810249223}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24545,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6839/4289194720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":4289194720},{"type":"v1","addr":"172.21.15.123:6839","nonce":4289194720}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[]} 2024-01-24T14:24:59.886 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.886+0000 7fca88342700 1 -- 172.21.15.19:0/731359826 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fca6c0a05f0 msgr2=0x7fca6c0a2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:59.886 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.886+0000 7fca88342700 1 --2- 172.21.15.19:0/731359826 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fca6c0a05f0 0x7fca6c0a2ab0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fca7c011550 tx=0x7fca7c009260 comp rx=0 tx=0).stop 2024-01-24T14:24:59.886 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.886+0000 7fca88342700 1 -- 172.21.15.19:0/731359826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca801040d0 msgr2=0x7fca800ff750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:24:59.886 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.886+0000 7fca88342700 1 --2- 172.21.15.19:0/731359826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca801040d0 0x7fca800ff750 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fca7000e9f0 tx=0x7fca7000ed00 comp rx=0 tx=0).stop 2024-01-24T14:24:59.887 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.886+0000 7fca88342700 1 -- 172.21.15.19:0/731359826 shutdown_connections 2024-01-24T14:24:59.887 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.886+0000 7fca88342700 1 --2- 172.21.15.19:0/731359826 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fca6c0a05f0 0x7fca6c0a2ab0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:59.887 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.886+0000 7fca88342700 1 --2- 172.21.15.19:0/731359826 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fca801017a0 0x7fca80101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:59.887 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.886+0000 7fca88342700 1 --2- 172.21.15.19:0/731359826 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fca80106a00 0x7fca800ffc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:59.887 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.886+0000 7fca88342700 1 --2- 172.21.15.19:0/731359826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fca801040d0 0x7fca800ff750 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:24:59.887 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.886+0000 7fca88342700 1 -- 172.21.15.19:0/731359826 >> 172.21.15.19:0/731359826 conn(0x7fca800fb170 msgr2=0x7fca80102410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:24:59.887 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.886+0000 7fca88342700 1 -- 172.21.15.19:0/731359826 shutdown_connections 2024-01-24T14:24:59.887 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:24:59.886+0000 7fca88342700 1 -- 172.21.15.19:0/731359826 wait complete. 2024-01-24T14:24:59.887 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 13 2024-01-24T14:24:59.900 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T14:25:00.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.063+0000 7fac5c679700 1 -- 172.21.15.19:0/162269557 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fac54101fc0 msgr2=0x7fac54102420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:00.064 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.063+0000 7fac5c679700 1 --2- 172.21.15.19:0/162269557 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fac54101fc0 0x7fac54102420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fac44009990 tx=0x7fac44009ca0 comp rx=0 tx=0).stop 2024-01-24T14:25:00.064 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.063+0000 7fac5c679700 1 -- 172.21.15.19:0/162269557 shutdown_connections 2024-01-24T14:25:00.064 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.063+0000 7fac5c679700 1 --2- 172.21.15.19:0/162269557 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac54102960 0x7fac54109210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:00.064 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.063+0000 7fac5c679700 1 --2- 172.21.15.19:0/162269557 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fac540feff0 0x7fac540ff410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:00.064 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.063+0000 7fac5c679700 1 --2- 172.21.15.19:0/162269557 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fac54101fc0 0x7fac54102420 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:00.064 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.063+0000 7fac5c679700 1 -- 172.21.15.19:0/162269557 >> 172.21.15.19:0/162269557 conn(0x7fac540fa900 msgr2=0x7fac540fcd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:00.064 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.063+0000 7fac5c679700 1 -- 172.21.15.19:0/162269557 shutdown_connections 2024-01-24T14:25:00.064 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.063+0000 7fac5c679700 1 -- 172.21.15.19:0/162269557 wait complete. 2024-01-24T14:25:00.066 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.064+0000 7fac5c679700 1 Processor -- start 2024-01-24T14:25:00.066 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.064+0000 7fac5c679700 1 -- start start 2024-01-24T14:25:00.067 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.064+0000 7fac5c679700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac540feff0 0x7fac54196a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:00.067 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.064+0000 7fac5c679700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fac54101fc0 0x7fac54196f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:00.067 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.064+0000 7fac5c679700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fac54102960 0x7fac5419e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:00.067 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.064+0000 7fac5c679700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5419e5d0 con 0x7fac54101fc0 2024-01-24T14:25:00.067 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.064+0000 7fac5c679700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5419e710 con 0x7fac540feff0 2024-01-24T14:25:00.067 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.064+0000 7fac5c679700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5419e8a0 con 0x7fac54102960 2024-01-24T14:25:00.067 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.064+0000 7fac59c14700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fac54101fc0 0x7fac54196f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:00.067 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.064+0000 7fac5a415700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac540feff0 0x7fac54196a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:00.067 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.065+0000 7fac5a415700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac540feff0 0x7fac54196a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.19:52334/0 (socket says 172.21.15.19:52334) 2024-01-24T14:25:00.067 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.065+0000 7fac5a415700 1 -- 172.21.15.19:0/308861814 learned_addr learned my addr 172.21.15.19:0/308861814 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:25:00.067 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.065+0000 7fac59c14700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fac54101fc0 0x7fac54196f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:54254/0 (socket says 172.21.15.19:54254) 2024-01-24T14:25:00.067 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.065+0000 7fac5ac16700 1 --2- 172.21.15.19:0/308861814 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fac54102960 0x7fac5419e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:00.067 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.065+0000 7fac5a415700 1 -- 172.21.15.19:0/308861814 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fac54102960 msgr2=0x7fac5419e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:00.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.065+0000 7fac5a415700 1 --2- 172.21.15.19:0/308861814 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fac54102960 0x7fac5419e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:00.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.065+0000 7fac5a415700 1 -- 172.21.15.19:0/308861814 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fac54101fc0 msgr2=0x7fac54196f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:00.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.065+0000 7fac5a415700 1 --2- 172.21.15.19:0/308861814 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fac54101fc0 0x7fac54196f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:00.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.065+0000 7fac5a415700 1 -- 172.21.15.19:0/308861814 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac440095f0 con 0x7fac540feff0 2024-01-24T14:25:00.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.065+0000 7fac5a415700 1 --2- 172.21.15.19:0/308861814 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac540feff0 0x7fac54196a50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fac5000e950 tx=0x7fac5000ec60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:00.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.065+0000 7fac4b7fe700 1 -- 172.21.15.19:0/308861814 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac50009770 con 0x7fac540feff0 2024-01-24T14:25:00.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.065+0000 7fac5c679700 1 -- 172.21.15.19:0/308861814 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac5419eb80 con 0x7fac540feff0 2024-01-24T14:25:00.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.065+0000 7fac4b7fe700 1 -- 172.21.15.19:0/308861814 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fac5000c7e0 con 0x7fac540feff0 2024-01-24T14:25:00.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.065+0000 7fac4b7fe700 1 -- 172.21.15.19:0/308861814 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac50003cf0 con 0x7fac540feff0 2024-01-24T14:25:00.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.065+0000 7fac5c679700 1 -- 172.21.15.19:0/308861814 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac5419f040 con 0x7fac540feff0 2024-01-24T14:25:00.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.066+0000 7fac5c679700 1 -- 172.21.15.19:0/308861814 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac5410c5a0 con 0x7fac540feff0 2024-01-24T14:25:00.069 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.068+0000 7fac4b7fe700 1 -- 172.21.15.19:0/308861814 <== mon.2 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7fac5000c950 con 0x7fac540feff0 2024-01-24T14:25:00.069 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.068+0000 7fac4b7fe700 1 --2- 172.21.15.19:0/308861814 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fac400a0570 0x7fac400a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:00.069 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.068+0000 7fac4b7fe700 1 -- 172.21.15.19:0/308861814 <== mon.2 v2:172.21.15.123:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fac50014070 con 0x7fac540feff0 2024-01-24T14:25:00.069 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.068+0000 7fac59c14700 1 --2- 172.21.15.19:0/308861814 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fac400a0570 0x7fac400a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:00.069 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.069+0000 7fac59c14700 1 --2- 172.21.15.19:0/308861814 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fac400a0570 0x7fac400a2a30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fac4400b2e0 tx=0x7fac44005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:00.071 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.071+0000 7fac4b7fe700 1 -- 172.21.15.19:0/308861814 <== mon.2 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fac50099220 con 0x7fac540feff0 2024-01-24T14:25:00.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:59 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:25:00.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:59 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:25:00.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:59 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:25:00.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:59 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:25:00.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:59 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:25:00.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:59 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:25:00.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:59 smithi194 ceph-mon[45450]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:25:00.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:59 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:25:00.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:59 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:25:00.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:59 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:25:00.095 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:24:59 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:25:00.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:59 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:25:00.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:59 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:25:00.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:59 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:25:00.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:59 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:25:00.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:59 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:25:00.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:59 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:25:00.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:59 smithi019 ceph-mon[40663]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:25:00.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:59 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:25:00.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:59 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:25:00.237 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:59 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:25:00.237 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:24:59 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:25:00.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.295+0000 7fac5c679700 1 -- 172.21.15.19:0/308861814 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac54107cd0 con 0x7fac540feff0 2024-01-24T14:25:00.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.296+0000 7fac4b7fe700 1 -- 172.21.15.19:0/308861814 <== mon.2 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) v1 ==== 74+0+18459 (secure 0 0 0) 0x7fac50098db0 con 0x7fac540feff0 2024-01-24T14:25:00.297 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:25:00.298 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":74,"fsid":"654b0c8e-bac3-11ee-95b1-87774f69a715","created":"2024-01-24T14:19:10.699369+0000","modified":"2024-01-24T14:24:31.040279+0000","last_up_change":"2024-01-24T14:24:30.032711+0000","last_in_change":"2024-01-24T14:24:13.776095+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T14:21:53.453429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"81380ec6-8e19-4eb4-8158-8fafaed8d7e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":2456786579},{"type":"v1","addr":"172.21.15.19:6803","nonce":2456786579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":2456786579},{"type":"v1","addr":"172.21.15.19:6805","nonce":2456786579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":2456786579},{"type":"v1","addr":"172.21.15.19:6809","nonce":2456786579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":2456786579},{"type":"v1","addr":"172.21.15.19:6807","nonce":2456786579}]},"public_addr":"172.21.15.19:6803/2456786579","cluster_addr":"172.21.15.19:6805/2456786579","heartbeat_back_addr":"172.21.15.19:6809/2456786579","heartbeat_front_addr":"172.21.15.19:6807/2456786579","state":["exists","up"]},{"osd":1,"uuid":"687f5036-a6d2-4aa1-86e7-7570e51ef179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":2407548889},{"type":"v1","addr":"172.21.15.123:6801","nonce":2407548889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":2407548889},{"type":"v1","addr":"172.21.15.123:6803","nonce":2407548889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":2407548889},{"type":"v1","addr":"172.21.15.123:6807","nonce":2407548889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":2407548889},{"type":"v1","addr":"172.21.15.123:6805","nonce":2407548889}]},"public_addr":"172.21.15.123:6801/2407548889","cluster_addr":"172.21.15.123:6803/2407548889","heartbeat_back_addr":"172.21.15.123:6807/2407548889","heartbeat_front_addr":"172.21.15.123:6805/2407548889","state":["exists","up"]},{"osd":2,"uuid":"3e810904-371d-4ca8-b43c-434957a2aa12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":3632698637},{"type":"v1","addr":"172.21.15.194:6801","nonce":3632698637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":3632698637},{"type":"v1","addr":"172.21.15.194:6803","nonce":3632698637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":3632698637},{"type":"v1","addr":"172.21.15.194:6807","nonce":3632698637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":3632698637},{"type":"v1","addr":"172.21.15.194:6805","nonce":3632698637}]},"public_addr":"172.21.15.194:6801/3632698637","cluster_addr":"172.21.15.194:6803/3632698637","heartbeat_back_addr":"172.21.15.194:6807/3632698637","heartbeat_front_addr":"172.21.15.194:6805/3632698637","state":["exists","up"]},{"osd":3,"uuid":"3665d6e0-010a-40bd-b0c6-682af56a0cbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":3868219159},{"type":"v1","addr":"172.21.15.19:6811","nonce":3868219159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3868219159},{"type":"v1","addr":"172.21.15.19:6813","nonce":3868219159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":3868219159},{"type":"v1","addr":"172.21.15.19:6817","nonce":3868219159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":3868219159},{"type":"v1","addr":"172.21.15.19:6815","nonce":3868219159}]},"public_addr":"172.21.15.19:6811/3868219159","cluster_addr":"172.21.15.19:6813/3868219159","heartbeat_back_addr":"172.21.15.19:6817/3868219159","heartbeat_front_addr":"172.21.15.19:6815/3868219159","state":["exists","up"]},{"osd":4,"uuid":"029fb28f-9af5-4239-8ee5-5ac5d04cd34b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":3611579809},{"type":"v1","addr":"172.21.15.123:6809","nonce":3611579809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":3611579809},{"type":"v1","addr":"172.21.15.123:6811","nonce":3611579809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":3611579809},{"type":"v1","addr":"172.21.15.123:6815","nonce":3611579809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":3611579809},{"type":"v1","addr":"172.21.15.123:6813","nonce":3611579809}]},"public_addr":"172.21.15.123:6809/3611579809","cluster_addr":"172.21.15.123:6811/3611579809","heartbeat_back_addr":"172.21.15.123:6815/3611579809","heartbeat_front_addr":"172.21.15.123:6813/3611579809","state":["exists","up"]},{"osd":5,"uuid":"6e97f23b-f803-4fed-a0a2-7aee7de4970f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":2483361871},{"type":"v1","addr":"172.21.15.194:6809","nonce":2483361871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":2483361871},{"type":"v1","addr":"172.21.15.194:6811","nonce":2483361871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":2483361871},{"type":"v1","addr":"172.21.15.194:6815","nonce":2483361871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":2483361871},{"type":"v1","addr":"172.21.15.194:6813","nonce":2483361871}]},"public_addr":"172.21.15.194:6809/2483361871","cluster_addr":"172.21.15.194:6811/2483361871","heartbeat_back_addr":"172.21.15.194:6815/2483361871","heartbeat_front_addr":"172.21.15.194:6813/2483361871","state":["exists","up"]},{"osd":6,"uuid":"c8c8ca2e-4cc6-4861-94b0-6211e4936c25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":1290513386},{"type":"v1","addr":"172.21.15.19:6819","nonce":1290513386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":1290513386},{"type":"v1","addr":"172.21.15.19:6821","nonce":1290513386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":1290513386},{"type":"v1","addr":"172.21.15.19:6825","nonce":1290513386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":1290513386},{"type":"v1","addr":"172.21.15.19:6823","nonce":1290513386}]},"public_addr":"172.21.15.19:6819/1290513386","cluster_addr":"172.21.15.19:6821/1290513386","heartbeat_back_addr":"172.21.15.19:6825/1290513386","heartbeat_front_addr":"172.21.15.19:6823/1290513386","state":["exists","up"]},{"osd":7,"uuid":"9aaea7a8-7bc3-4e32-9328-20a6c30a6d8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1391966446},{"type":"v1","addr":"172.21.15.123:6817","nonce":1391966446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1391966446},{"type":"v1","addr":"172.21.15.123:6819","nonce":1391966446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1391966446},{"type":"v1","addr":"172.21.15.123:6823","nonce":1391966446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1391966446},{"type":"v1","addr":"172.21.15.123:6821","nonce":1391966446}]},"public_addr":"172.21.15.123:6817/1391966446","cluster_addr":"172.21.15.123:6819/1391966446","heartbeat_back_addr":"172.21.15.123:6823/1391966446","heartbeat_front_addr":"172.21.15.123:6821/1391966446","state":["exists","up"]},{"osd":8,"uuid":"e918ec1b-7608-4f73-9067-29082ea63999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":2260577139},{"type":"v1","addr":"172.21.15.194:6817","nonce":2260577139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":2260577139},{"type":"v1","addr":"172.21.15.194:6819","nonce":2260577139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":2260577139},{"type":"v1","addr":"172.21.15.194:6823","nonce":2260577139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":2260577139},{"type":"v1","addr":"172.21.15.194:6821","nonce":2260577139}]},"public_addr":"172.21.15.194:6817/2260577139","cluster_addr":"172.21.15.194:6819/2260577139","heartbeat_back_addr":"172.21.15.194:6823/2260577139","heartbeat_front_addr":"172.21.15.194:6821/2260577139","state":["exists","up"]},{"osd":9,"uuid":"0fd76722-3319-4828-ab68-108607a4bd0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":4004843075},{"type":"v1","addr":"172.21.15.19:6827","nonce":4004843075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":4004843075},{"type":"v1","addr":"172.21.15.19:6829","nonce":4004843075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":4004843075},{"type":"v1","addr":"172.21.15.19:6833","nonce":4004843075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":4004843075},{"type":"v1","addr":"172.21.15.19:6831","nonce":4004843075}]},"public_addr":"172.21.15.19:6827/4004843075","cluster_addr":"172.21.15.19:6829/4004843075","heartbeat_back_addr":"172.21.15.19:6833/4004843075","heartbeat_front_addr":"172.21.15.19:6831/4004843075","state":["exists","up"]},{"osd":10,"uuid":"624bb8f5-6d72-43d5-9986-1b48b8a564ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2591845068},{"type":"v1","addr":"172.21.15.123:6825","nonce":2591845068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":2591845068},{"type":"v1","addr":"172.21.15.123:6827","nonce":2591845068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2591845068},{"type":"v1","addr":"172.21.15.123:6831","nonce":2591845068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2591845068},{"type":"v1","addr":"172.21.15.123:6829","nonce":2591845068}]},"public_addr":"172.21.15.123:6825/2591845068","cluster_addr":"172.21.15.123:6827/2591845068","heartbeat_back_addr":"172.21.15.123:6831/2591845068","heartbeat_front_addr":"172.21.15.123:6829/2591845068","state":["exists","up"]},{"osd":11,"uuid":"219a55b3-b01e-471d-90da-1a3043bf3e34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":3985496260},{"type":"v1","addr":"172.21.15.194:6825","nonce":3985496260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":3985496260},{"type":"v1","addr":"172.21.15.194:6827","nonce":3985496260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":3985496260},{"type":"v1","addr":"172.21.15.194:6831","nonce":3985496260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":3985496260},{"type":"v1","addr":"172.21.15.194:6829","nonce":3985496260}]},"public_addr":"172.21.15.194:6825/3985496260","cluster_addr":"172.21.15.194:6827/3985496260","heartbeat_back_addr":"172.21.15.194:6831/3985496260","heartbeat_front_addr":"172.21.15.194:6829/3985496260","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:21:17.278347+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:21:34.469282+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:21:51.139919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:22:09.126458+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:22:26.186801+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:22:43.178997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:23:01.109570+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:23:18.592191+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:23:35.685046+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:23:53.574820+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:24:10.987531+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:24:28.231206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.19:0/1047252857":"2024-01-25T14:20:02.911218+0000","172.21.15.19:6801/3758061162":"2024-01-25T14:20:02.911218+0000","172.21.15.19:6800/3758061162":"2024-01-25T14:20:02.911218+0000","172.21.15.19:6801/2754937713":"2024-01-25T14:19:39.853212+0000","172.21.15.19:0/4062420470":"2024-01-25T14:19:39.853212+0000","172.21.15.19:6800/2754937713":"2024-01-25T14:19:39.853212+0000","172.21.15.19:0/822613692":"2024-01-25T14:20:02.911218+0000","172.21.15.19:0/3970666946":"2024-01-25T14:19:39.853212+0000","172.21.15.19:0/1116053938":"2024-01-25T14:19:39.853212+0000","172.21.15.19:0/1456770388":"2024-01-25T14:20:02.911218+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T14:25:00.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.299+0000 7fac5c679700 1 -- 172.21.15.19:0/308861814 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fac400a0570 msgr2=0x7fac400a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:00.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.299+0000 7fac5c679700 1 --2- 172.21.15.19:0/308861814 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fac400a0570 0x7fac400a2a30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fac4400b2e0 tx=0x7fac44005fb0 comp rx=0 tx=0).stop 2024-01-24T14:25:00.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.299+0000 7fac5c679700 1 -- 172.21.15.19:0/308861814 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac540feff0 msgr2=0x7fac54196a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:00.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.299+0000 7fac5c679700 1 --2- 172.21.15.19:0/308861814 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac540feff0 0x7fac54196a50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fac5000e950 tx=0x7fac5000ec60 comp rx=0 tx=0).stop 2024-01-24T14:25:00.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.299+0000 7fac5c679700 1 -- 172.21.15.19:0/308861814 shutdown_connections 2024-01-24T14:25:00.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.299+0000 7fac5c679700 1 --2- 172.21.15.19:0/308861814 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fac54102960 0x7fac5419e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:00.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.299+0000 7fac5c679700 1 --2- 172.21.15.19:0/308861814 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fac540feff0 0x7fac54196a50 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:00.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.299+0000 7fac5c679700 1 --2- 172.21.15.19:0/308861814 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fac400a0570 0x7fac400a2a30 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:00.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.299+0000 7fac5c679700 1 --2- 172.21.15.19:0/308861814 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fac54101fc0 0x7fac54196f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:00.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.299+0000 7fac5c679700 1 -- 172.21.15.19:0/308861814 >> 172.21.15.19:0/308861814 conn(0x7fac540fa900 msgr2=0x7fac540fcba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:00.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.299+0000 7fac5c679700 1 -- 172.21.15.19:0/308861814 shutdown_connections 2024-01-24T14:25:00.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.299+0000 7fac5c679700 1 -- 172.21.15.19:0/308861814 wait complete. 2024-01-24T14:25:00.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:59 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:25:00.301 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:59 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:25:00.301 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:59 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:25:00.301 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:59 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:25:00.301 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:59 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:25:00.301 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:59 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:25:00.301 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:59 smithi123 ceph-mon[44966]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:25:00.301 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:59 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:25:00.301 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:59 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:25:00.301 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:59 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:25:00.301 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:24:59 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-24T14:25:00.315 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T14:21:53.453429+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-24T14:25:00.316 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T14:25:00.474 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.474+0000 7f4a52c39700 1 -- 172.21.15.19:0/157442964 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4a4c104140 msgr2=0x7f4a4c106530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:00.474 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.474+0000 7f4a52c39700 1 --2- 172.21.15.19:0/157442964 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4a4c104140 0x7f4a4c106530 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4a38009990 tx=0x7f4a38009ca0 comp rx=0 tx=0).stop 2024-01-24T14:25:00.474 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.474+0000 7f4a52c39700 1 -- 172.21.15.19:0/157442964 shutdown_connections 2024-01-24T14:25:00.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.474+0000 7f4a52c39700 1 --2- 172.21.15.19:0/157442964 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a4c106a70 0x7f4a4c108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:00.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.474+0000 7f4a52c39700 1 --2- 172.21.15.19:0/157442964 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4a4c101810 0x7f4a4c103c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:00.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.474+0000 7f4a52c39700 1 --2- 172.21.15.19:0/157442964 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4a4c104140 0x7f4a4c106530 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:00.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.474+0000 7f4a52c39700 1 -- 172.21.15.19:0/157442964 >> 172.21.15.19:0/157442964 conn(0x7f4a4c0fb1a0 msgr2=0x7f4a4c0fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:00.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.474+0000 7f4a52c39700 1 -- 172.21.15.19:0/157442964 shutdown_connections 2024-01-24T14:25:00.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.474+0000 7f4a52c39700 1 -- 172.21.15.19:0/157442964 wait complete. 2024-01-24T14:25:00.477 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.475+0000 7f4a52c39700 1 Processor -- start 2024-01-24T14:25:00.477 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.475+0000 7f4a52c39700 1 -- start start 2024-01-24T14:25:00.477 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.475+0000 7f4a52c39700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4a4c101810 0x7f4a4c101180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:00.477 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.475+0000 7f4a52c39700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4a4c104140 0x7f4a4c0ff7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:00.477 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.475+0000 7f4a52c39700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a4c106a70 0x7f4a4c0ffd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:00.477 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.475+0000 7f4a52c39700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4c1004a0 con 0x7f4a4c104140 2024-01-24T14:25:00.477 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.475+0000 7f4a52c39700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4c100610 con 0x7f4a4c106a70 2024-01-24T14:25:00.477 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.475+0000 7f4a52c39700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4c1a0aa0 con 0x7f4a4c101810 2024-01-24T14:25:00.478 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.475+0000 7f4a4bfff700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4a4c104140 0x7f4a4c0ff7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:00.478 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.475+0000 7f4a4bfff700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4a4c104140 0x7f4a4c0ff7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:54272/0 (socket says 172.21.15.19:54272) 2024-01-24T14:25:00.478 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.475+0000 7f4a4bfff700 1 -- 172.21.15.19:0/4213548703 learned_addr learned my addr 172.21.15.19:0/4213548703 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:25:00.478 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.475+0000 7f4a509d5700 1 --2- 172.21.15.19:0/4213548703 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4a4c101810 0x7f4a4c101180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:00.478 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.475+0000 7f4a511d6700 1 --2- 172.21.15.19:0/4213548703 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a4c106a70 0x7f4a4c0ffd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:00.478 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.476+0000 7f4a509d5700 1 -- 172.21.15.19:0/4213548703 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a4c106a70 msgr2=0x7f4a4c0ffd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:00.478 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.476+0000 7f4a509d5700 1 --2- 172.21.15.19:0/4213548703 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a4c106a70 0x7f4a4c0ffd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:00.478 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.476+0000 7f4a509d5700 1 -- 172.21.15.19:0/4213548703 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4a4c104140 msgr2=0x7f4a4c0ff7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:00.478 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.476+0000 7f4a509d5700 1 --2- 172.21.15.19:0/4213548703 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4a4c104140 0x7f4a4c0ff7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:00.478 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.476+0000 7f4a509d5700 1 -- 172.21.15.19:0/4213548703 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a380095f0 con 0x7f4a4c101810 2024-01-24T14:25:00.478 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.476+0000 7f4a4bfff700 1 --2- 172.21.15.19:0/4213548703 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4a4c104140 0x7f4a4c0ff7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T14:25:00.478 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.476+0000 7f4a509d5700 1 --2- 172.21.15.19:0/4213548703 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4a4c101810 0x7f4a4c101180 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4a4000e9f0 tx=0x7f4a4000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:00.478 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.476+0000 7f4a49ffb700 1 -- 172.21.15.19:0/4213548703 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a4000c9c0 con 0x7f4a4c101810 2024-01-24T14:25:00.479 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.476+0000 7f4a52c39700 1 -- 172.21.15.19:0/4213548703 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a4c1a0d80 con 0x7f4a4c101810 2024-01-24T14:25:00.479 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.476+0000 7f4a49ffb700 1 -- 172.21.15.19:0/4213548703 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4a40010430 con 0x7f4a4c101810 2024-01-24T14:25:00.479 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.476+0000 7f4a49ffb700 1 -- 172.21.15.19:0/4213548703 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a40018680 con 0x7f4a4c101810 2024-01-24T14:25:00.479 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.476+0000 7f4a52c39700 1 -- 172.21.15.19:0/4213548703 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a4c1a11d0 con 0x7f4a4c101810 2024-01-24T14:25:00.479 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.477+0000 7f4a52c39700 1 -- 172.21.15.19:0/4213548703 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a4c10c570 con 0x7f4a4c101810 2024-01-24T14:25:00.479 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.479+0000 7f4a49ffb700 1 -- 172.21.15.19:0/4213548703 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f4a40010860 con 0x7f4a4c101810 2024-01-24T14:25:00.480 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.479+0000 7f4a49ffb700 1 --2- 172.21.15.19:0/4213548703 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f4a3c0a0520 0x7f4a3c0a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:00.480 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.479+0000 7f4a4bfff700 1 --2- 172.21.15.19:0/4213548703 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f4a3c0a0520 0x7f4a3c0a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:00.480 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.479+0000 7f4a49ffb700 1 -- 172.21.15.19:0/4213548703 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f4a40014070 con 0x7f4a4c101810 2024-01-24T14:25:00.480 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.480+0000 7f4a4bfff700 1 --2- 172.21.15.19:0/4213548703 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f4a3c0a0520 0x7f4a3c0a29e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4a3800b2e0 tx=0x7f4a38005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:00.486 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.486+0000 7f4a49ffb700 1 -- 172.21.15.19:0/4213548703 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f4a40095830 con 0x7f4a4c101810 2024-01-24T14:25:00.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.751+0000 7f4a52c39700 1 -- 172.21.15.19:0/4213548703 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4a4c1a17e0 con 0x7f4a4c101810 2024-01-24T14:25:00.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.752+0000 7f4a49ffb700 1 -- 172.21.15.19:0/4213548703 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v74) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4a40098e50 con 0x7f4a4c101810 2024-01-24T14:25:00.752 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 1 2024-01-24T14:25:00.754 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.754+0000 7f4a52c39700 1 -- 172.21.15.19:0/4213548703 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f4a3c0a0520 msgr2=0x7f4a3c0a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:00.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.754+0000 7f4a52c39700 1 --2- 172.21.15.19:0/4213548703 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f4a3c0a0520 0x7f4a3c0a29e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4a3800b2e0 tx=0x7f4a38005fb0 comp rx=0 tx=0).stop 2024-01-24T14:25:00.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.754+0000 7f4a52c39700 1 -- 172.21.15.19:0/4213548703 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4a4c101810 msgr2=0x7f4a4c101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:00.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.754+0000 7f4a52c39700 1 --2- 172.21.15.19:0/4213548703 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4a4c101810 0x7f4a4c101180 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4a4000e9f0 tx=0x7f4a4000ed00 comp rx=0 tx=0).stop 2024-01-24T14:25:00.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.754+0000 7f4a52c39700 1 -- 172.21.15.19:0/4213548703 shutdown_connections 2024-01-24T14:25:00.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.754+0000 7f4a52c39700 1 --2- 172.21.15.19:0/4213548703 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a4c106a70 0x7f4a4c0ffd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:00.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.754+0000 7f4a52c39700 1 --2- 172.21.15.19:0/4213548703 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4a4c101810 0x7f4a4c101180 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:00.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.754+0000 7f4a52c39700 1 --2- 172.21.15.19:0/4213548703 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f4a3c0a0520 0x7f4a3c0a29e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:00.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.754+0000 7f4a52c39700 1 --2- 172.21.15.19:0/4213548703 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4a4c104140 0x7f4a4c0ff7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:00.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.754+0000 7f4a52c39700 1 -- 172.21.15.19:0/4213548703 >> 172.21.15.19:0/4213548703 conn(0x7f4a4c0fb1a0 msgr2=0x7f4a4c102cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:00.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.754+0000 7f4a52c39700 1 -- 172.21.15.19:0/4213548703 shutdown_connections 2024-01-24T14:25:00.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.754+0000 7f4a52c39700 1 -- 172.21.15.19:0/4213548703 wait complete. 2024-01-24T14:25:00.769 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-24T14:25:00.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.908+0000 7f2b58c28700 1 -- 172.21.15.19:0/3420273122 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2b541017b0 msgr2=0x7f2b54108060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:00.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.908+0000 7f2b58c28700 1 --2- 172.21.15.19:0/3420273122 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2b541017b0 0x7f2b54108060 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c00b0d0 tx=0x7f2b4c00b3e0 comp rx=0 tx=0).stop 2024-01-24T14:25:00.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.908+0000 7f2b58c28700 1 -- 172.21.15.19:0/3420273122 shutdown_connections 2024-01-24T14:25:00.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.908+0000 7f2b58c28700 1 --2- 172.21.15.19:0/3420273122 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2b541017b0 0x7f2b54108060 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:00.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.908+0000 7f2b58c28700 1 --2- 172.21.15.19:0/3420273122 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2b540ffc20 0x7f2b54100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:00.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.908+0000 7f2b58c28700 1 --2- 172.21.15.19:0/3420273122 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b54100e10 0x7f2b54101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:00.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.908+0000 7f2b58c28700 1 -- 172.21.15.19:0/3420273122 >> 172.21.15.19:0/3420273122 conn(0x7f2b540fb170 msgr2=0x7f2b540fd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:00.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.908+0000 7f2b58c28700 1 -- 172.21.15.19:0/3420273122 shutdown_connections 2024-01-24T14:25:00.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.908+0000 7f2b58c28700 1 -- 172.21.15.19:0/3420273122 wait complete. 2024-01-24T14:25:00.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.909+0000 7f2b58c28700 1 Processor -- start 2024-01-24T14:25:00.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.909+0000 7f2b58c28700 1 -- start start 2024-01-24T14:25:00.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.909+0000 7f2b58c28700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2b540ffc20 0x7f2b54196af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:00.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.909+0000 7f2b58c28700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b54100e10 0x7f2b54197030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:00.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.909+0000 7f2b58c28700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2b541017b0 0x7f2b5419e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:00.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.909+0000 7f2b58c28700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5419e670 con 0x7f2b540ffc20 2024-01-24T14:25:00.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.909+0000 7f2b58c28700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5419e7b0 con 0x7f2b54100e10 2024-01-24T14:25:00.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.909+0000 7f2b58c28700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5419e940 con 0x7f2b541017b0 2024-01-24T14:25:00.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.909+0000 7f2b5259c700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2b540ffc20 0x7f2b54196af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:00.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.909+0000 7f2b5259c700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2b540ffc20 0x7f2b54196af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:54294/0 (socket says 172.21.15.19:54294) 2024-01-24T14:25:00.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.909+0000 7f2b5259c700 1 -- 172.21.15.19:0/593904488 learned_addr learned my addr 172.21.15.19:0/593904488 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:25:00.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.909+0000 7f2b52d9d700 1 --2- 172.21.15.19:0/593904488 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2b541017b0 0x7f2b5419e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:00.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.909+0000 7f2b51d9b700 1 --2- 172.21.15.19:0/593904488 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b54100e10 0x7f2b54197030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:00.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.909+0000 7f2b5259c700 1 -- 172.21.15.19:0/593904488 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2b541017b0 msgr2=0x7f2b5419e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:00.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.909+0000 7f2b5259c700 1 --2- 172.21.15.19:0/593904488 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2b541017b0 0x7f2b5419e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:00.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.909+0000 7f2b5259c700 1 -- 172.21.15.19:0/593904488 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b54100e10 msgr2=0x7f2b54197030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:00.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.909+0000 7f2b5259c700 1 --2- 172.21.15.19:0/593904488 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b54100e10 0x7f2b54197030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:00.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.909+0000 7f2b5259c700 1 -- 172.21.15.19:0/593904488 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b3c009520 con 0x7f2b540ffc20 2024-01-24T14:25:00.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.910+0000 7f2b5259c700 1 --2- 172.21.15.19:0/593904488 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2b540ffc20 0x7f2b54196af0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2b4400d980 tx=0x7f2b4400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:00.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.910+0000 7f2b4b7fe700 1 -- 172.21.15.19:0/593904488 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b44004020 con 0x7f2b540ffc20 2024-01-24T14:25:00.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.910+0000 7f2b4b7fe700 1 -- 172.21.15.19:0/593904488 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2b4400f690 con 0x7f2b540ffc20 2024-01-24T14:25:00.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.910+0000 7f2b58c28700 1 -- 172.21.15.19:0/593904488 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b4c009d00 con 0x7f2b540ffc20 2024-01-24T14:25:00.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.910+0000 7f2b4b7fe700 1 -- 172.21.15.19:0/593904488 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b44003a60 con 0x7f2b540ffc20 2024-01-24T14:25:00.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.910+0000 7f2b58c28700 1 -- 172.21.15.19:0/593904488 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b5419eff0 con 0x7f2b540ffc20 2024-01-24T14:25:00.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.911+0000 7f2b58c28700 1 -- 172.21.15.19:0/593904488 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b54107d00 con 0x7f2b540ffc20 2024-01-24T14:25:00.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.912+0000 7f2b4b7fe700 1 -- 172.21.15.19:0/593904488 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f2b4400fb00 con 0x7f2b540ffc20 2024-01-24T14:25:00.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.913+0000 7f2b4b7fe700 1 --2- 172.21.15.19:0/593904488 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f2b400a05f0 0x7f2b400a2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:00.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.913+0000 7f2b51d9b700 1 --2- 172.21.15.19:0/593904488 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f2b400a05f0 0x7f2b400a2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:00.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.913+0000 7f2b4b7fe700 1 -- 172.21.15.19:0/593904488 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f2b440cb3e0 con 0x7f2b540ffc20 2024-01-24T14:25:00.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.914+0000 7f2b51d9b700 1 --2- 172.21.15.19:0/593904488 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f2b400a05f0 0x7f2b400a2ab0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c009df0 tx=0x7f2b3c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:00.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:00.917+0000 7f2b4b7fe700 1 -- 172.21.15.19:0/593904488 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f2b44095440 con 0x7f2b540ffc20 2024-01-24T14:25:01.135 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.135+0000 7f2b58c28700 1 -- 172.21.15.19:0/593904488 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f2b5402cd60 con 0x7f2b540ffc20 2024-01-24T14:25:01.135 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.135+0000 7f2b4b7fe700 1 -- 172.21.15.19:0/593904488 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v74) v1 ==== 84+0+65 (secure 0 0 0) 0x7f2b44098a60 con 0x7f2b540ffc20 2024-01-24T14:25:01.135 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:25:01.135 INFO:teuthology.orchestra.run.smithi019.stdout:[ 2024-01-24T14:25:01.136 INFO:teuthology.orchestra.run.smithi019.stdout: { 2024-01-24T14:25:01.136 INFO:teuthology.orchestra.run.smithi019.stdout: "poolnum": 1, 2024-01-24T14:25:01.136 INFO:teuthology.orchestra.run.smithi019.stdout: "poolname": ".mgr" 2024-01-24T14:25:01.136 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-01-24T14:25:01.136 INFO:teuthology.orchestra.run.smithi019.stdout:] 2024-01-24T14:25:01.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.137+0000 7f2b58c28700 1 -- 172.21.15.19:0/593904488 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f2b400a05f0 msgr2=0x7f2b400a2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:01.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.137+0000 7f2b58c28700 1 --2- 172.21.15.19:0/593904488 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f2b400a05f0 0x7f2b400a2ab0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c009df0 tx=0x7f2b3c009260 comp rx=0 tx=0).stop 2024-01-24T14:25:01.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.137+0000 7f2b58c28700 1 -- 172.21.15.19:0/593904488 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2b540ffc20 msgr2=0x7f2b54196af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:01.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.137+0000 7f2b58c28700 1 --2- 172.21.15.19:0/593904488 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2b540ffc20 0x7f2b54196af0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2b4400d980 tx=0x7f2b4400dc90 comp rx=0 tx=0).stop 2024-01-24T14:25:01.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.137+0000 7f2b58c28700 1 -- 172.21.15.19:0/593904488 shutdown_connections 2024-01-24T14:25:01.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.137+0000 7f2b58c28700 1 --2- 172.21.15.19:0/593904488 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f2b400a05f0 0x7f2b400a2ab0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:01.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.137+0000 7f2b58c28700 1 --2- 172.21.15.19:0/593904488 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2b540ffc20 0x7f2b54196af0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:01.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.137+0000 7f2b58c28700 1 --2- 172.21.15.19:0/593904488 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2b541017b0 0x7f2b5419e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:01.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.137+0000 7f2b58c28700 1 --2- 172.21.15.19:0/593904488 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b54100e10 0x7f2b54197030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:01.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.137+0000 7f2b58c28700 1 -- 172.21.15.19:0/593904488 >> 172.21.15.19:0/593904488 conn(0x7f2b540fb170 msgr2=0x7f2b540fd390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:01.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.138+0000 7f2b58c28700 1 -- 172.21.15.19:0/593904488 shutdown_connections 2024-01-24T14:25:01.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.138+0000 7f2b58c28700 1 -- 172.21.15.19:0/593904488 wait complete. 2024-01-24T14:25:01.150 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:00 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/731359826' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T14:25:01.150 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:00 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/308861814' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T14:25:01.150 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:00 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:25:01.151 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:00 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:25:01.151 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:00 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:25:01.151 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:00 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:25:01.151 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:00 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:25:01.151 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:00 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:25:01.151 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:00 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:25:01.151 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:00 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:25:01.151 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:00 smithi019 ceph-mon[40663]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:25:01.151 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:00 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/4213548703' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-01-24T14:25:01.152 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-24T14:25:01.152 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-24T14:25:01.294 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.293+0000 7fc540ab5700 1 -- 172.21.15.19:0/238280768 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc53c100920 msgr2=0x7fc53c100da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:01.294 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.293+0000 7fc540ab5700 1 --2- 172.21.15.19:0/238280768 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc53c100920 0x7fc53c100da0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc53400b0d0 tx=0x7fc53400b3e0 comp rx=0 tx=0).stop 2024-01-24T14:25:01.294 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.293+0000 7fc540ab5700 1 -- 172.21.15.19:0/238280768 shutdown_connections 2024-01-24T14:25:01.294 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.293+0000 7fc540ab5700 1 --2- 172.21.15.19:0/238280768 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc53c100920 0x7fc53c100da0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:01.294 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.293+0000 7fc540ab5700 1 --2- 172.21.15.19:0/238280768 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc53c0ff600 0x7fc53c0ffa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:01.294 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.293+0000 7fc540ab5700 1 --2- 172.21.15.19:0/238280768 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc53c0fff60 0x7fc53c1003e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:01.294 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.293+0000 7fc540ab5700 1 -- 172.21.15.19:0/238280768 >> 172.21.15.19:0/238280768 conn(0x7fc53c0fb1a0 msgr2=0x7fc53c0fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:01.294 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.293+0000 7fc540ab5700 1 -- 172.21.15.19:0/238280768 shutdown_connections 2024-01-24T14:25:01.294 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.293+0000 7fc540ab5700 1 -- 172.21.15.19:0/238280768 wait complete. 2024-01-24T14:25:01.294 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.294+0000 7fc540ab5700 1 Processor -- start 2024-01-24T14:25:01.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.294+0000 7fc540ab5700 1 -- start start 2024-01-24T14:25:01.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.294+0000 7fc540ab5700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc53c0ff600 0x7fc53c105180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:01.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.294+0000 7fc540ab5700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc53c0fff60 0x7fc53c1056c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:01.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.294+0000 7fc540ab5700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc53c100920 0x7fc53c101d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:01.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.294+0000 7fc540ab5700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc53c1024b0 con 0x7fc53c100920 2024-01-24T14:25:01.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.294+0000 7fc540ab5700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc53c102620 con 0x7fc53c0fff60 2024-01-24T14:25:01.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.294+0000 7fc540ab5700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc53c1027e0 con 0x7fc53c0ff600 2024-01-24T14:25:01.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.295+0000 7fc53ad9d700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc53c100920 0x7fc53c101d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:01.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.295+0000 7fc53ad9d700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc53c100920 0x7fc53c101d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:54318/0 (socket says 172.21.15.19:54318) 2024-01-24T14:25:01.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.295+0000 7fc53ad9d700 1 -- 172.21.15.19:0/1454001198 learned_addr learned my addr 172.21.15.19:0/1454001198 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:25:01.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.295+0000 7fc53a59c700 1 --2- 172.21.15.19:0/1454001198 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc53c0ff600 0x7fc53c105180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:01.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.295+0000 7fc531d9b700 1 --2- 172.21.15.19:0/1454001198 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc53c0fff60 0x7fc53c1056c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:01.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.295+0000 7fc53ad9d700 1 -- 172.21.15.19:0/1454001198 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc53c0ff600 msgr2=0x7fc53c105180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:01.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.295+0000 7fc53ad9d700 1 --2- 172.21.15.19:0/1454001198 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc53c0ff600 0x7fc53c105180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:01.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.295+0000 7fc53ad9d700 1 -- 172.21.15.19:0/1454001198 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc53c0fff60 msgr2=0x7fc53c1056c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:01.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.295+0000 7fc53ad9d700 1 --2- 172.21.15.19:0/1454001198 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc53c0fff60 0x7fc53c1056c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:01.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.295+0000 7fc53ad9d700 1 -- 172.21.15.19:0/1454001198 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc534009d00 con 0x7fc53c100920 2024-01-24T14:25:01.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.295+0000 7fc531d9b700 1 --2- 172.21.15.19:0/1454001198 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc53c0fff60 0x7fc53c1056c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T14:25:01.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.295+0000 7fc53ad9d700 1 --2- 172.21.15.19:0/1454001198 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc53c100920 0x7fc53c101d10 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc534007a00 tx=0x7fc5340094c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:01.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.295+0000 7fc533fff700 1 -- 172.21.15.19:0/1454001198 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc53400e040 con 0x7fc53c100920 2024-01-24T14:25:01.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.295+0000 7fc533fff700 1 -- 172.21.15.19:0/1454001198 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc534003e10 con 0x7fc53c100920 2024-01-24T14:25:01.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.295+0000 7fc533fff700 1 -- 172.21.15.19:0/1454001198 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc534012a90 con 0x7fc53c100920 2024-01-24T14:25:01.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.295+0000 7fc540ab5700 1 -- 172.21.15.19:0/1454001198 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc53c1a5190 con 0x7fc53c100920 2024-01-24T14:25:01.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.295+0000 7fc540ab5700 1 -- 172.21.15.19:0/1454001198 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc53c1a55b0 con 0x7fc53c100920 2024-01-24T14:25:01.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.296+0000 7fc540ab5700 1 -- 172.21.15.19:0/1454001198 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc53c10a3e0 con 0x7fc53c100920 2024-01-24T14:25:01.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:00 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/731359826' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T14:25:01.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:00 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/308861814' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T14:25:01.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:00 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:25:01.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:00 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:25:01.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:00 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:25:01.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:00 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:25:01.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:00 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:25:01.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:00 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:25:01.301 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:00 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:25:01.301 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:00 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:25:01.301 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:00 smithi123 ceph-mon[44966]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:25:01.301 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:00 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/4213548703' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-01-24T14:25:01.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.298+0000 7fc533fff700 1 -- 172.21.15.19:0/1454001198 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7fc5340056c0 con 0x7fc53c100920 2024-01-24T14:25:01.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.298+0000 7fc533fff700 1 --2- 172.21.15.19:0/1454001198 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fc5280a05a0 0x7fc5280a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:01.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.299+0000 7fc53a59c700 1 --2- 172.21.15.19:0/1454001198 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fc5280a05a0 0x7fc5280a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:01.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.299+0000 7fc533fff700 1 -- 172.21.15.19:0/1454001198 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc5340ccb00 con 0x7fc53c100920 2024-01-24T14:25:01.302 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.299+0000 7fc53a59c700 1 --2- 172.21.15.19:0/1454001198 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fc5280a05a0 0x7fc5280a2a60 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc52c007c60 tx=0x7fc52c0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:01.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.303+0000 7fc533fff700 1 -- 172.21.15.19:0/1454001198 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fc534096bd0 con 0x7fc53c100920 2024-01-24T14:25:01.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:00 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/731359826' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T14:25:01.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:00 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/308861814' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T14:25:01.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:00 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:25:01.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:00 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:25:01.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:00 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:25:01.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:00 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:25:01.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:00 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:25:01.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:00 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:25:01.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:00 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-24T14:25:01.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:00 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-01-24T14:25:01.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:00 smithi194 ceph-mon[45450]: from='mgr.14152 172.21.15.19:0/397492189' entity='mgr.x' 2024-01-24T14:25:01.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:00 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/4213548703' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-01-24T14:25:01.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:01.518+0000 7fc540ab5700 1 -- 172.21.15.19:0/1454001198 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fc53c02cc70 con 0x7fc53c100920 2024-01-24T14:25:02.021 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.021+0000 7fc533fff700 1 -- 172.21.15.19:0/1454001198 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v75) v1 ==== 138+0+0 (secure 0 0 0) 0x7fc53409a1f0 con 0x7fc53c100920 2024-01-24T14:25:02.027 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.025+0000 7fc540ab5700 1 -- 172.21.15.19:0/1454001198 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fc5280a05a0 msgr2=0x7fc5280a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:02.027 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.025+0000 7fc540ab5700 1 --2- 172.21.15.19:0/1454001198 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fc5280a05a0 0x7fc5280a2a60 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc52c007c60 tx=0x7fc52c0073d0 comp rx=0 tx=0).stop 2024-01-24T14:25:02.028 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.025+0000 7fc540ab5700 1 -- 172.21.15.19:0/1454001198 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc53c100920 msgr2=0x7fc53c101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:02.028 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.025+0000 7fc540ab5700 1 --2- 172.21.15.19:0/1454001198 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc53c100920 0x7fc53c101d10 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc534007a00 tx=0x7fc5340094c0 comp rx=0 tx=0).stop 2024-01-24T14:25:02.028 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.026+0000 7fc540ab5700 1 -- 172.21.15.19:0/1454001198 shutdown_connections 2024-01-24T14:25:02.028 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.026+0000 7fc540ab5700 1 --2- 172.21.15.19:0/1454001198 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fc5280a05a0 0x7fc5280a2a60 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:02.028 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.026+0000 7fc540ab5700 1 --2- 172.21.15.19:0/1454001198 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc53c100920 0x7fc53c101d10 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:02.028 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.026+0000 7fc540ab5700 1 --2- 172.21.15.19:0/1454001198 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc53c0ff600 0x7fc53c105180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:02.028 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.026+0000 7fc540ab5700 1 --2- 172.21.15.19:0/1454001198 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc53c0fff60 0x7fc53c1056c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:02.028 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.026+0000 7fc540ab5700 1 -- 172.21.15.19:0/1454001198 >> 172.21.15.19:0/1454001198 conn(0x7fc53c0fb1a0 msgr2=0x7fc53c0fcd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:02.029 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.028+0000 7fc540ab5700 1 -- 172.21.15.19:0/1454001198 shutdown_connections 2024-01-24T14:25:02.029 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.028+0000 7fc540ab5700 1 -- 172.21.15.19:0/1454001198 wait complete. 2024-01-24T14:25:02.029 INFO:teuthology.orchestra.run.smithi019.stderr:pool 'cephfs_metadata' created 2024-01-24T14:25:02.043 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-01-24T14:25:02.192 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.189+0000 7f28c31be700 1 -- 172.21.15.19:0/1416367196 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f28bc0ffc40 msgr2=0x7f28bc100060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:02.192 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.189+0000 7f28c31be700 1 --2- 172.21.15.19:0/1416367196 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f28bc0ffc40 0x7f28bc100060 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f28b00099c0 tx=0x7f28b0009cd0 comp rx=0 tx=0).stop 2024-01-24T14:25:02.193 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.190+0000 7f28c31be700 1 -- 172.21.15.19:0/1416367196 shutdown_connections 2024-01-24T14:25:02.193 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.190+0000 7f28c31be700 1 --2- 172.21.15.19:0/1416367196 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f28bc0ffc40 0x7f28bc100060 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:02.193 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.190+0000 7f28c31be700 1 --2- 172.21.15.19:0/1416367196 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28bc1017d0 0x7f28bc108080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:02.193 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.190+0000 7f28c31be700 1 --2- 172.21.15.19:0/1416367196 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f28bc100e30 0x7f28bc101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:02.193 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.190+0000 7f28c31be700 1 -- 172.21.15.19:0/1416367196 >> 172.21.15.19:0/1416367196 conn(0x7f28bc0fb1d0 msgr2=0x7f28bc0fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:02.193 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.190+0000 7f28c31be700 1 -- 172.21.15.19:0/1416367196 shutdown_connections 2024-01-24T14:25:02.193 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.190+0000 7f28c31be700 1 -- 172.21.15.19:0/1416367196 wait complete. 2024-01-24T14:25:02.193 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.190+0000 7f28c31be700 1 Processor -- start 2024-01-24T14:25:02.193 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.190+0000 7f28c31be700 1 -- start start 2024-01-24T14:25:02.193 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.190+0000 7f28c31be700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f28bc0ffc40 0x7f28bc196a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:02.193 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.190+0000 7f28c31be700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28bc100e30 0x7f28bc196f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:02.193 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.190+0000 7f28c31be700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f28bc1017d0 0x7f28bc19dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:02.193 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.190+0000 7f28c31be700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28bc19e5b0 con 0x7f28bc0ffc40 2024-01-24T14:25:02.194 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.190+0000 7f28c31be700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28bc19e6f0 con 0x7f28bc100e30 2024-01-24T14:25:02.194 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.190+0000 7f28c31be700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28bc19e880 con 0x7f28bc1017d0 2024-01-24T14:25:02.194 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.191+0000 7f28c0f5a700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f28bc0ffc40 0x7f28bc196a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:02.194 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.191+0000 7f28c0f5a700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f28bc0ffc40 0x7f28bc196a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:60972/0 (socket says 172.21.15.19:60972) 2024-01-24T14:25:02.194 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.191+0000 7f28c0f5a700 1 -- 172.21.15.19:0/4131764027 learned_addr learned my addr 172.21.15.19:0/4131764027 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:25:02.194 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.191+0000 7f28bbfff700 1 --2- 172.21.15.19:0/4131764027 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28bc100e30 0x7f28bc196f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:02.194 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.191+0000 7f28c175b700 1 --2- 172.21.15.19:0/4131764027 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f28bc1017d0 0x7f28bc19dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:02.194 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.191+0000 7f28c0f5a700 1 -- 172.21.15.19:0/4131764027 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f28bc1017d0 msgr2=0x7f28bc19dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:02.194 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.191+0000 7f28c0f5a700 1 --2- 172.21.15.19:0/4131764027 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f28bc1017d0 0x7f28bc19dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:02.194 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.191+0000 7f28c0f5a700 1 -- 172.21.15.19:0/4131764027 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28bc100e30 msgr2=0x7f28bc196f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:02.194 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.191+0000 7f28c0f5a700 1 --2- 172.21.15.19:0/4131764027 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28bc100e30 0x7f28bc196f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:02.194 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.191+0000 7f28c0f5a700 1 -- 172.21.15.19:0/4131764027 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28b00095f0 con 0x7f28bc0ffc40 2024-01-24T14:25:02.194 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.191+0000 7f28c0f5a700 1 --2- 172.21.15.19:0/4131764027 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f28bc0ffc40 0x7f28bc196a30 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f28b00058e0 tx=0x7f28b000f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:02.195 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.191+0000 7f28b9ffb700 1 -- 172.21.15.19:0/4131764027 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b001d070 con 0x7f28bc0ffc40 2024-01-24T14:25:02.195 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.191+0000 7f28b9ffb700 1 -- 172.21.15.19:0/4131764027 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f28b0005510 con 0x7f28bc0ffc40 2024-01-24T14:25:02.195 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.191+0000 7f28c31be700 1 -- 172.21.15.19:0/4131764027 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28bc19eb00 con 0x7f28bc0ffc40 2024-01-24T14:25:02.195 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.191+0000 7f28b9ffb700 1 -- 172.21.15.19:0/4131764027 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b0004bb0 con 0x7f28bc0ffc40 2024-01-24T14:25:02.195 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.192+0000 7f28c31be700 1 -- 172.21.15.19:0/4131764027 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28bc19ef20 con 0x7f28bc0ffc40 2024-01-24T14:25:02.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.193+0000 7f28c31be700 1 -- 172.21.15.19:0/4131764027 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28bc10b450 con 0x7f28bc0ffc40 2024-01-24T14:25:02.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.194+0000 7f28b9ffb700 1 -- 172.21.15.19:0/4131764027 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f28b0004d30 con 0x7f28bc0ffc40 2024-01-24T14:25:02.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.195+0000 7f28b9ffb700 1 --2- 172.21.15.19:0/4131764027 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f28ac0a0520 0x7f28ac0a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:02.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.195+0000 7f28bbfff700 1 --2- 172.21.15.19:0/4131764027 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f28ac0a0520 0x7f28ac0a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:02.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.195+0000 7f28b9ffb700 1 -- 172.21.15.19:0/4131764027 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f28b00cc9f0 con 0x7f28bc0ffc40 2024-01-24T14:25:02.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.196+0000 7f28bbfff700 1 --2- 172.21.15.19:0/4131764027 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f28ac0a0520 0x7f28ac0a29e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f28a8009bf0 tx=0x7f28a8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:02.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.199+0000 7f28b9ffb700 1 -- 172.21.15.19:0/4131764027 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f28b00968b0 con 0x7f28bc0ffc40 2024-01-24T14:25:02.235 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:01 smithi019 ceph-mon[40663]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:25:02.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:01 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/593904488' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-01-24T14:25:02.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:01 smithi123 ceph-mon[44966]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:25:02.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:01 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/593904488' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-01-24T14:25:02.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:01 smithi194 ceph-mon[45450]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:25:02.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:01 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/593904488' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-01-24T14:25:02.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:02.415+0000 7f28c31be700 1 -- 172.21.15.19:0/4131764027 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f28bc19f850 con 0x7f28bc0ffc40 2024-01-24T14:25:03.031 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.028+0000 7f28b9ffb700 1 -- 172.21.15.19:0/4131764027 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v76) v1 ==== 171+0+0 (secure 0 0 0) 0x7f28b0099ed0 con 0x7f28bc0ffc40 2024-01-24T14:25:03.037 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.037+0000 7f28c31be700 1 -- 172.21.15.19:0/4131764027 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f28ac0a0520 msgr2=0x7f28ac0a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:03.038 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.038+0000 7f28c31be700 1 --2- 172.21.15.19:0/4131764027 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f28ac0a0520 0x7f28ac0a29e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f28a8009bf0 tx=0x7f28a8009260 comp rx=0 tx=0).stop 2024-01-24T14:25:03.038 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.038+0000 7f28c31be700 1 -- 172.21.15.19:0/4131764027 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f28bc0ffc40 msgr2=0x7f28bc196a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:03.038 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.038+0000 7f28c31be700 1 --2- 172.21.15.19:0/4131764027 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f28bc0ffc40 0x7f28bc196a30 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f28b00058e0 tx=0x7f28b000f740 comp rx=0 tx=0).stop 2024-01-24T14:25:03.041 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.041+0000 7f28c31be700 1 -- 172.21.15.19:0/4131764027 shutdown_connections 2024-01-24T14:25:03.041 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.041+0000 7f28c31be700 1 --2- 172.21.15.19:0/4131764027 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f28ac0a0520 0x7f28ac0a29e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:03.041 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.041+0000 7f28c31be700 1 --2- 172.21.15.19:0/4131764027 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f28bc0ffc40 0x7f28bc196a30 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:03.041 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.041+0000 7f28c31be700 1 --2- 172.21.15.19:0/4131764027 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f28bc1017d0 0x7f28bc19dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:03.042 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.041+0000 7f28c31be700 1 --2- 172.21.15.19:0/4131764027 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28bc100e30 0x7f28bc196f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:03.042 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.041+0000 7f28c31be700 1 -- 172.21.15.19:0/4131764027 >> 172.21.15.19:0/4131764027 conn(0x7f28bc0fb1d0 msgr2=0x7f28bc068ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:03.045 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.045+0000 7f28c31be700 1 -- 172.21.15.19:0/4131764027 shutdown_connections 2024-01-24T14:25:03.047 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.047+0000 7f28c31be700 1 -- 172.21.15.19:0/4131764027 wait complete. 2024-01-24T14:25:03.051 INFO:teuthology.orchestra.run.smithi019.stderr:pool 'cephfs_data' created 2024-01-24T14:25:03.066 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-24T14:25:03.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.209+0000 7f13e56a1700 1 -- 172.21.15.19:0/419306427 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f13e0101850 msgr2=0x7f13e0103c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:03.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.209+0000 7f13e56a1700 1 --2- 172.21.15.19:0/419306427 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f13e0101850 0x7f13e0103c40 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f13d0009990 tx=0x7f13d0009ca0 comp rx=0 tx=0).stop 2024-01-24T14:25:03.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.209+0000 7f13e56a1700 1 -- 172.21.15.19:0/419306427 shutdown_connections 2024-01-24T14:25:03.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.209+0000 7f13e56a1700 1 --2- 172.21.15.19:0/419306427 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f13e0101850 0x7f13e0103c40 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:03.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.209+0000 7f13e56a1700 1 --2- 172.21.15.19:0/419306427 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13e0106ab0 0x7f13e0108ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:03.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.209+0000 7f13e56a1700 1 --2- 172.21.15.19:0/419306427 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13e0104180 0x7f13e0106570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:03.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.209+0000 7f13e56a1700 1 -- 172.21.15.19:0/419306427 >> 172.21.15.19:0/419306427 conn(0x7f13e00fb200 msgr2=0x7f13e00fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:03.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.209+0000 7f13e56a1700 1 -- 172.21.15.19:0/419306427 shutdown_connections 2024-01-24T14:25:03.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.209+0000 7f13e56a1700 1 -- 172.21.15.19:0/419306427 wait complete. 2024-01-24T14:25:03.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.210+0000 7f13e56a1700 1 Processor -- start 2024-01-24T14:25:03.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.210+0000 7f13e56a1700 1 -- start start 2024-01-24T14:25:03.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.210+0000 7f13e56a1700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f13e0101850 0x7f13e0196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:03.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.210+0000 7f13e56a1700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13e0104180 0x7f13e0196f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:03.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.210+0000 7f13e56a1700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13e0106ab0 0x7f13e019dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:03.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.210+0000 7f13e56a1700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13e019e590 con 0x7f13e0101850 2024-01-24T14:25:03.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.210+0000 7f13e56a1700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13e019e6d0 con 0x7f13e0106ab0 2024-01-24T14:25:03.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.210+0000 7f13e56a1700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13e019e860 con 0x7f13e0104180 2024-01-24T14:25:03.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.210+0000 7f13deffd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f13e0101850 0x7f13e0196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:03.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.210+0000 7f13deffd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f13e0101850 0x7f13e0196a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:32770/0 (socket says 172.21.15.19:32770) 2024-01-24T14:25:03.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.210+0000 7f13deffd700 1 -- 172.21.15.19:0/2825508584 learned_addr learned my addr 172.21.15.19:0/2825508584 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:25:03.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.211+0000 7f13df7fe700 1 --2- 172.21.15.19:0/2825508584 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13e0106ab0 0x7f13e019dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:03.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.211+0000 7f13de7fc700 1 --2- 172.21.15.19:0/2825508584 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13e0104180 0x7f13e0196f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:03.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.211+0000 7f13deffd700 1 -- 172.21.15.19:0/2825508584 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13e0104180 msgr2=0x7f13e0196f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:03.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.211+0000 7f13deffd700 1 --2- 172.21.15.19:0/2825508584 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13e0104180 0x7f13e0196f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:03.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.211+0000 7f13deffd700 1 -- 172.21.15.19:0/2825508584 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13e0106ab0 msgr2=0x7f13e019dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:03.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.211+0000 7f13deffd700 1 --2- 172.21.15.19:0/2825508584 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13e0106ab0 0x7f13e019dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:03.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.211+0000 7f13deffd700 1 -- 172.21.15.19:0/2825508584 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d00095f0 con 0x7f13e0101850 2024-01-24T14:25:03.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.211+0000 7f13de7fc700 1 --2- 172.21.15.19:0/2825508584 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13e0104180 0x7f13e0196f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T14:25:03.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.211+0000 7f13df7fe700 1 --2- 172.21.15.19:0/2825508584 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13e0106ab0 0x7f13e019dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T14:25:03.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.211+0000 7f13deffd700 1 --2- 172.21.15.19:0/2825508584 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f13e0101850 0x7f13e0196a10 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f13d0000c00 tx=0x7f13d000f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:03.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.211+0000 7f13c7fff700 1 -- 172.21.15.19:0/2825508584 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d001d070 con 0x7f13e0101850 2024-01-24T14:25:03.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.211+0000 7f13c7fff700 1 -- 172.21.15.19:0/2825508584 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f13d000fc00 con 0x7f13e0101850 2024-01-24T14:25:03.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.211+0000 7f13e56a1700 1 -- 172.21.15.19:0/2825508584 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13e019eae0 con 0x7f13e0101850 2024-01-24T14:25:03.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.211+0000 7f13c7fff700 1 -- 172.21.15.19:0/2825508584 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d0017650 con 0x7f13e0101850 2024-01-24T14:25:03.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.211+0000 7f13e56a1700 1 -- 172.21.15.19:0/2825508584 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13e019ef90 con 0x7f13e0101850 2024-01-24T14:25:03.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.212+0000 7f13e56a1700 1 -- 172.21.15.19:0/2825508584 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13e00fcd80 con 0x7f13e0101850 2024-01-24T14:25:03.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.214+0000 7f13c7fff700 1 -- 172.21.15.19:0/2825508584 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f13d0022470 con 0x7f13e0101850 2024-01-24T14:25:03.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.214+0000 7f13c7fff700 1 --2- 172.21.15.19:0/2825508584 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f13c80a0520 0x7f13c80a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:03.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.215+0000 7f13de7fc700 1 --2- 172.21.15.19:0/2825508584 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f13c80a0520 0x7f13c80a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:03.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.215+0000 7f13c7fff700 1 -- 172.21.15.19:0/2825508584 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7879+0+0 (secure 0 0 0) 0x7f13d00ccda0 con 0x7f13e0101850 2024-01-24T14:25:03.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.215+0000 7f13de7fc700 1 --2- 172.21.15.19:0/2825508584 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f13c80a0520 0x7f13c80a29e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f13d4005fd0 tx=0x7f13d4005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:03.223 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.219+0000 7f13c7fff700 1 -- 172.21.15.19:0/2825508584 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f13d0096b80 con 0x7f13e0101850 2024-01-24T14:25:03.235 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:02 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/1454001198' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-01-24T14:25:03.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:02 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/1454001198' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-01-24T14:25:03.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:02 smithi019 ceph-mon[40663]: osdmap e75: 12 total, 12 up, 12 in 2024-01-24T14:25:03.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:02 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/4131764027' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-01-24T14:25:03.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:02 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/1454001198' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-01-24T14:25:03.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:02 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/1454001198' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-01-24T14:25:03.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:02 smithi123 ceph-mon[44966]: osdmap e75: 12 total, 12 up, 12 in 2024-01-24T14:25:03.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:02 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/4131764027' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-01-24T14:25:03.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:02 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/1454001198' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-01-24T14:25:03.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:02 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/1454001198' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-01-24T14:25:03.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:02 smithi194 ceph-mon[45450]: osdmap e75: 12 total, 12 up, 12 in 2024-01-24T14:25:03.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:02 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/4131764027' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-01-24T14:25:03.482 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.482+0000 7f13e56a1700 1 -- 172.21.15.19:0/2825508584 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f13e002cc70 con 0x7f13e0101850 2024-01-24T14:25:03.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.491+0000 7f13c7fff700 1 -- 172.21.15.19:0/2825508584 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7f13d009a1a0 con 0x7f13e0101850 2024-01-24T14:25:03.497 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.497+0000 7f13e56a1700 1 -- 172.21.15.19:0/2825508584 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f13c80a0520 msgr2=0x7f13c80a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:03.497 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.497+0000 7f13e56a1700 1 --2- 172.21.15.19:0/2825508584 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f13c80a0520 0x7f13c80a29e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f13d4005fd0 tx=0x7f13d4005ee0 comp rx=0 tx=0).stop 2024-01-24T14:25:03.497 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.497+0000 7f13e56a1700 1 -- 172.21.15.19:0/2825508584 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f13e0101850 msgr2=0x7f13e0196a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:03.497 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.497+0000 7f13e56a1700 1 --2- 172.21.15.19:0/2825508584 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f13e0101850 0x7f13e0196a10 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f13d0000c00 tx=0x7f13d000f710 comp rx=0 tx=0).stop 2024-01-24T14:25:03.497 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.497+0000 7f13e56a1700 1 -- 172.21.15.19:0/2825508584 shutdown_connections 2024-01-24T14:25:03.497 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.497+0000 7f13e56a1700 1 --2- 172.21.15.19:0/2825508584 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f13c80a0520 0x7f13c80a29e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:03.497 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.497+0000 7f13e56a1700 1 --2- 172.21.15.19:0/2825508584 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f13e0101850 0x7f13e0196a10 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:03.498 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.497+0000 7f13e56a1700 1 --2- 172.21.15.19:0/2825508584 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13e0106ab0 0x7f13e019dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:03.498 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.497+0000 7f13e56a1700 1 --2- 172.21.15.19:0/2825508584 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13e0104180 0x7f13e0196f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:03.498 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.497+0000 7f13e56a1700 1 -- 172.21.15.19:0/2825508584 >> 172.21.15.19:0/2825508584 conn(0x7f13e00fb200 msgr2=0x7f13e01024c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:03.498 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.498+0000 7f13e56a1700 1 -- 172.21.15.19:0/2825508584 shutdown_connections 2024-01-24T14:25:03.498 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.499+0000 7f13e56a1700 1 -- 172.21.15.19:0/2825508584 wait complete. 2024-01-24T14:25:03.510 INFO:teuthology.orchestra.run.smithi019.stderr:new fs with metadata pool 2 and data pool 3 2024-01-24T14:25:03.523 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-01-24T14:25:03.523 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-01-24T14:25:03.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.665+0000 7f981db22700 1 -- 172.21.15.19:0/2745158544 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9818100e20 msgr2=0x7f9818101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:03.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.665+0000 7f981db22700 1 --2- 172.21.15.19:0/2745158544 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9818100e20 0x7f9818101280 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f98000099c0 tx=0x7f9800009cd0 comp rx=0 tx=0).stop 2024-01-24T14:25:03.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.665+0000 7f981db22700 1 -- 172.21.15.19:0/2745158544 shutdown_connections 2024-01-24T14:25:03.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.665+0000 7f981db22700 1 --2- 172.21.15.19:0/2745158544 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98181017c0 0x7f9818108070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:03.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.665+0000 7f981db22700 1 --2- 172.21.15.19:0/2745158544 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f98180ffc30 0x7f9818100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:03.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.665+0000 7f981db22700 1 --2- 172.21.15.19:0/2745158544 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9818100e20 0x7f9818101280 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:03.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.665+0000 7f981db22700 1 -- 172.21.15.19:0/2745158544 >> 172.21.15.19:0/2745158544 conn(0x7f98180fb1a0 msgr2=0x7f98180fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:03.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.665+0000 7f981db22700 1 -- 172.21.15.19:0/2745158544 shutdown_connections 2024-01-24T14:25:03.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.665+0000 7f981db22700 1 -- 172.21.15.19:0/2745158544 wait complete. 2024-01-24T14:25:03.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.666+0000 7f981db22700 1 Processor -- start 2024-01-24T14:25:03.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.666+0000 7f981db22700 1 -- start start 2024-01-24T14:25:03.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.666+0000 7f981db22700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98180ffc30 0x7f9818196a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:03.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.666+0000 7f981db22700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9818100e20 0x7f9818196fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:03.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.666+0000 7f981db22700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f98181017c0 0x7f981819e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:03.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.666+0000 7f981db22700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981819e5e0 con 0x7f9818100e20 2024-01-24T14:25:03.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.666+0000 7f981db22700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981819e720 con 0x7f98180ffc30 2024-01-24T14:25:03.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.666+0000 7f981db22700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981819e8b0 con 0x7f98181017c0 2024-01-24T14:25:03.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.666+0000 7f9816ffd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9818100e20 0x7f9818196fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:03.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.666+0000 7f9816ffd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9818100e20 0x7f9818196fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:32790/0 (socket says 172.21.15.19:32790) 2024-01-24T14:25:03.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.666+0000 7f9816ffd700 1 -- 172.21.15.19:0/1153400139 learned_addr learned my addr 172.21.15.19:0/1153400139 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:25:03.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.666+0000 7f98177fe700 1 --2- 172.21.15.19:0/1153400139 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98180ffc30 0x7f9818196a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:03.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.667+0000 7f9817fff700 1 --2- 172.21.15.19:0/1153400139 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f98181017c0 0x7f981819e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:03.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.667+0000 7f9816ffd700 1 -- 172.21.15.19:0/1153400139 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f98181017c0 msgr2=0x7f981819e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:03.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.667+0000 7f9816ffd700 1 --2- 172.21.15.19:0/1153400139 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f98181017c0 0x7f981819e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:03.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.667+0000 7f9816ffd700 1 -- 172.21.15.19:0/1153400139 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98180ffc30 msgr2=0x7f9818196a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:03.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.667+0000 7f9816ffd700 1 --2- 172.21.15.19:0/1153400139 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98180ffc30 0x7f9818196a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:03.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.667+0000 7f9816ffd700 1 -- 172.21.15.19:0/1153400139 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98000095f0 con 0x7f9818100e20 2024-01-24T14:25:03.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.667+0000 7f9817fff700 1 --2- 172.21.15.19:0/1153400139 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f98181017c0 0x7f981819e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T14:25:03.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.667+0000 7f98177fe700 1 --2- 172.21.15.19:0/1153400139 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98180ffc30 0x7f9818196a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T14:25:03.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.667+0000 7f9816ffd700 1 --2- 172.21.15.19:0/1153400139 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9818100e20 0x7f9818196fa0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9800009990 tx=0x7f980000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:03.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.667+0000 7f9814ff9700 1 -- 172.21.15.19:0/1153400139 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980001d070 con 0x7f9818100e20 2024-01-24T14:25:03.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.667+0000 7f9814ff9700 1 -- 172.21.15.19:0/1153400139 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f98000054f0 con 0x7f9818100e20 2024-01-24T14:25:03.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.667+0000 7f9814ff9700 1 -- 172.21.15.19:0/1153400139 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9800004bd0 con 0x7f9818100e20 2024-01-24T14:25:03.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.667+0000 7f981db22700 1 -- 172.21.15.19:0/1153400139 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f981819eb30 con 0x7f9818100e20 2024-01-24T14:25:03.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.667+0000 7f981db22700 1 -- 172.21.15.19:0/1153400139 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f981819ef00 con 0x7f9818100e20 2024-01-24T14:25:03.671 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.668+0000 7f980e7fc700 1 -- 172.21.15.19:0/1153400139 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97fc0052f0 con 0x7f9818100e20 2024-01-24T14:25:03.671 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.669+0000 7f9814ff9700 1 -- 172.21.15.19:0/1153400139 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f980000fb80 con 0x7f9818100e20 2024-01-24T14:25:03.671 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.670+0000 7f9814ff9700 1 --2- 172.21.15.19:0/1153400139 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f98040a0520 0x7f98040a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:03.671 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.670+0000 7f9814ff9700 1 -- 172.21.15.19:0/1153400139 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f98000cc740 con 0x7f9818100e20 2024-01-24T14:25:03.671 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.670+0000 7f98177fe700 1 --2- 172.21.15.19:0/1153400139 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f98040a0520 0x7f98040a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:03.671 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.671+0000 7f98177fe700 1 --2- 172.21.15.19:0/1153400139 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f98040a0520 0x7f98040a29e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9818100c80 tx=0x7f980800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:03.675 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.675+0000 7f9814ff9700 1 -- 172.21.15.19:0/1153400139 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f9800096420 con 0x7f9818100e20 2024-01-24T14:25:03.896 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:03.896+0000 7f980e7fc700 1 -- 172.21.15.19:0/1153400139 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7f97fc005160 con 0x7f9818100e20 2024-01-24T14:25:04.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:04 smithi123 ceph-mon[44966]: pgmap v215: 33 pgs: 6 creating+peering, 1 active+clean, 26 unknown; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:25:04.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:04 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/4131764027' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-01-24T14:25:04.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:04 smithi123 ceph-mon[44966]: osdmap e76: 12 total, 12 up, 12 in 2024-01-24T14:25:04.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:04 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/2825508584' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-01-24T14:25:04.299 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:04 smithi123 ceph-mon[44966]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-01-24T14:25:04.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:04 smithi123 ceph-mon[44966]: osdmap e77: 12 total, 12 up, 12 in 2024-01-24T14:25:04.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:04 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/2825508584' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-01-24T14:25:04.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:04 smithi123 ceph-mon[44966]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-01-24T14:25:04.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:04 smithi123 ceph-mon[44966]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-01-24T14:25:04.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:04 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/1153400139' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-01-24T14:25:04.300 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:04 smithi123 ceph-mon[44966]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-01-24T14:25:04.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:04 smithi194 ceph-mon[45450]: pgmap v215: 33 pgs: 6 creating+peering, 1 active+clean, 26 unknown; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:25:04.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:04 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/4131764027' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-01-24T14:25:04.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:04 smithi194 ceph-mon[45450]: osdmap e76: 12 total, 12 up, 12 in 2024-01-24T14:25:04.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:04 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/2825508584' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-01-24T14:25:04.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:04 smithi194 ceph-mon[45450]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-01-24T14:25:04.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:04 smithi194 ceph-mon[45450]: osdmap e77: 12 total, 12 up, 12 in 2024-01-24T14:25:04.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:04 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/2825508584' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-01-24T14:25:04.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:04 smithi194 ceph-mon[45450]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-01-24T14:25:04.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:04 smithi194 ceph-mon[45450]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-01-24T14:25:04.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:04 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/1153400139' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-01-24T14:25:04.345 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:04 smithi194 ceph-mon[45450]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-01-24T14:25:04.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:04 smithi019 ceph-mon[40663]: pgmap v215: 33 pgs: 6 creating+peering, 1 active+clean, 26 unknown; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:25:04.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:04 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/4131764027' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-01-24T14:25:04.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:04 smithi019 ceph-mon[40663]: osdmap e76: 12 total, 12 up, 12 in 2024-01-24T14:25:04.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:04 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/2825508584' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-01-24T14:25:04.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:04 smithi019 ceph-mon[40663]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-01-24T14:25:04.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:04 smithi019 ceph-mon[40663]: osdmap e77: 12 total, 12 up, 12 in 2024-01-24T14:25:04.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:04 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/2825508584' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-01-24T14:25:04.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:04 smithi019 ceph-mon[40663]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-01-24T14:25:04.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:04 smithi019 ceph-mon[40663]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-01-24T14:25:04.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:04 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/1153400139' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-01-24T14:25:04.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:04 smithi019 ceph-mon[40663]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-01-24T14:25:04.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.490+0000 7f9814ff9700 1 -- 172.21.15.19:0/1153400139 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v78) v1 ==== 157+0+0 (secure 0 0 0) 0x7f9800099a40 con 0x7f9818100e20 2024-01-24T14:25:04.493 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.493+0000 7f980e7fc700 1 -- 172.21.15.19:0/1153400139 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f98040a0520 msgr2=0x7f98040a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:04.494 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.493+0000 7f980e7fc700 1 --2- 172.21.15.19:0/1153400139 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f98040a0520 0x7f98040a29e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9818100c80 tx=0x7f980800b410 comp rx=0 tx=0).stop 2024-01-24T14:25:04.494 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.493+0000 7f980e7fc700 1 -- 172.21.15.19:0/1153400139 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9818100e20 msgr2=0x7f9818196fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:04.494 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.493+0000 7f980e7fc700 1 --2- 172.21.15.19:0/1153400139 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9818100e20 0x7f9818196fa0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9800009990 tx=0x7f980000f690 comp rx=0 tx=0).stop 2024-01-24T14:25:04.494 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.493+0000 7f980e7fc700 1 -- 172.21.15.19:0/1153400139 shutdown_connections 2024-01-24T14:25:04.494 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.493+0000 7f980e7fc700 1 --2- 172.21.15.19:0/1153400139 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f98181017c0 0x7f981819e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:04.494 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.493+0000 7f980e7fc700 1 --2- 172.21.15.19:0/1153400139 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f98180ffc30 0x7f9818196a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:04.494 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.493+0000 7f980e7fc700 1 --2- 172.21.15.19:0/1153400139 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f98040a0520 0x7f98040a29e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:04.494 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.493+0000 7f980e7fc700 1 --2- 172.21.15.19:0/1153400139 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9818100e20 0x7f9818196fa0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:04.494 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.493+0000 7f980e7fc700 1 -- 172.21.15.19:0/1153400139 >> 172.21.15.19:0/1153400139 conn(0x7f98180fb1a0 msgr2=0x7f9818067ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:04.494 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.494+0000 7f980e7fc700 1 -- 172.21.15.19:0/1153400139 shutdown_connections 2024-01-24T14:25:04.494 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.494+0000 7f980e7fc700 1 -- 172.21.15.19:0/1153400139 wait complete. 2024-01-24T14:25:04.509 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-01-24T14:25:04.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.664+0000 7f6738589700 1 -- 172.21.15.19:0/3186602775 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67301017c0 msgr2=0x7f6730108070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:04.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.664+0000 7f6738589700 1 --2- 172.21.15.19:0/3186602775 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67301017c0 0x7f6730108070 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f672800b0d0 tx=0x7f672800b3e0 comp rx=0 tx=0).stop 2024-01-24T14:25:04.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.664+0000 7f6738589700 1 -- 172.21.15.19:0/3186602775 shutdown_connections 2024-01-24T14:25:04.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.664+0000 7f6738589700 1 --2- 172.21.15.19:0/3186602775 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67301017c0 0x7f6730108070 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:04.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.664+0000 7f6738589700 1 --2- 172.21.15.19:0/3186602775 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f67300ffc30 0x7f6730100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:04.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.664+0000 7f6738589700 1 --2- 172.21.15.19:0/3186602775 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6730100e20 0x7f6730101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:04.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.664+0000 7f6738589700 1 -- 172.21.15.19:0/3186602775 >> 172.21.15.19:0/3186602775 conn(0x7f67300fb1a0 msgr2=0x7f67300fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:04.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.664+0000 7f6738589700 1 -- 172.21.15.19:0/3186602775 shutdown_connections 2024-01-24T14:25:04.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.665+0000 7f6738589700 1 -- 172.21.15.19:0/3186602775 wait complete. 2024-01-24T14:25:04.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.665+0000 7f6738589700 1 Processor -- start 2024-01-24T14:25:04.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.665+0000 7f6738589700 1 -- start start 2024-01-24T14:25:04.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.665+0000 7f6738589700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f67300ffc30 0x7f6730198cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:04.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.665+0000 7f6738589700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6730100e20 0x7f6730199230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:04.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.665+0000 7f6738589700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67301017c0 0x7f67301a0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:04.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.665+0000 7f6738589700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67301a0830 con 0x7f6730100e20 2024-01-24T14:25:04.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.665+0000 7f6738589700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67301a09a0 con 0x7f67301017c0 2024-01-24T14:25:04.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.665+0000 7f6738589700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67301a0b60 con 0x7f67300ffc30 2024-01-24T14:25:04.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.665+0000 7f6735b24700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6730100e20 0x7f6730199230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:04.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.665+0000 7f6735b24700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6730100e20 0x7f6730199230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:32814/0 (socket says 172.21.15.19:32814) 2024-01-24T14:25:04.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.665+0000 7f6735b24700 1 -- 172.21.15.19:0/1220630885 learned_addr learned my addr 172.21.15.19:0/1220630885 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:25:04.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.666+0000 7f6736b26700 1 --2- 172.21.15.19:0/1220630885 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67301017c0 0x7f67301a0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:04.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.666+0000 7f6736325700 1 --2- 172.21.15.19:0/1220630885 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f67300ffc30 0x7f6730198cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:04.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.666+0000 7f6735b24700 1 -- 172.21.15.19:0/1220630885 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f67300ffc30 msgr2=0x7f6730198cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:04.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.666+0000 7f6735b24700 1 --2- 172.21.15.19:0/1220630885 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f67300ffc30 0x7f6730198cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:04.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.666+0000 7f6735b24700 1 -- 172.21.15.19:0/1220630885 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67301017c0 msgr2=0x7f67301a0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:04.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.666+0000 7f6735b24700 1 --2- 172.21.15.19:0/1220630885 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67301017c0 0x7f67301a0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:04.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.666+0000 7f6735b24700 1 -- 172.21.15.19:0/1220630885 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6728009d00 con 0x7f6730100e20 2024-01-24T14:25:04.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.666+0000 7f6736b26700 1 --2- 172.21.15.19:0/1220630885 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67301017c0 0x7f67301a0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T14:25:04.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.666+0000 7f6735b24700 1 --2- 172.21.15.19:0/1220630885 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6730100e20 0x7f6730199230 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f672000e9a0 tx=0x7f672000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:04.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.666+0000 7f67277fe700 1 -- 172.21.15.19:0/1220630885 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6720009790 con 0x7f6730100e20 2024-01-24T14:25:04.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.666+0000 7f67277fe700 1 -- 172.21.15.19:0/1220630885 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f672000cbd0 con 0x7f6730100e20 2024-01-24T14:25:04.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.666+0000 7f67277fe700 1 -- 172.21.15.19:0/1220630885 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6720018680 con 0x7f6730100e20 2024-01-24T14:25:04.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.666+0000 7f6738589700 1 -- 172.21.15.19:0/1220630885 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67301a0e40 con 0x7f6730100e20 2024-01-24T14:25:04.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.666+0000 7f6738589700 1 -- 172.21.15.19:0/1220630885 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67301a1260 con 0x7f6730100e20 2024-01-24T14:25:04.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.667+0000 7f6738589700 1 -- 172.21.15.19:0/1220630885 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f673010b450 con 0x7f6730100e20 2024-01-24T14:25:04.671 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.668+0000 7f67277fe700 1 -- 172.21.15.19:0/1220630885 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f672000cd40 con 0x7f6730100e20 2024-01-24T14:25:04.671 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.669+0000 7f67277fe700 1 --2- 172.21.15.19:0/1220630885 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f671c0a0640 0x7f671c0a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:04.671 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.669+0000 7f6736325700 1 --2- 172.21.15.19:0/1220630885 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f671c0a0640 0x7f671c0a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:04.671 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.669+0000 7f67277fe700 1 -- 172.21.15.19:0/1220630885 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8205+0+0 (secure 0 0 0) 0x7f6720014070 con 0x7f6730100e20 2024-01-24T14:25:04.671 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.670+0000 7f6736325700 1 --2- 172.21.15.19:0/1220630885 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f671c0a0640 0x7f671c0a2b00 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f672c005950 tx=0x7f672c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:04.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.673+0000 7f67277fe700 1 -- 172.21.15.19:0/1220630885 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f6720095c20 con 0x7f6730100e20 2024-01-24T14:25:04.886 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:04.886+0000 7f6738589700 1 -- 172.21.15.19:0/1220630885 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f673010b610 con 0x7f6730100e20 2024-01-24T14:25:05.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:05 smithi019 ceph-mon[40663]: mds.? [v2:172.21.15.123:6838/4289194720,v1:172.21.15.123:6839/4289194720] up:active 2024-01-24T14:25:05.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:05 smithi019 ceph-mon[40663]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-01-24T14:25:05.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:05 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/1153400139' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-01-24T14:25:05.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:05 smithi019 ceph-mon[40663]: osdmap e78: 12 total, 12 up, 12 in 2024-01-24T14:25:05.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:05 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/1220630885' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-01-24T14:25:05.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:05 smithi123 ceph-mon[44966]: mds.? [v2:172.21.15.123:6838/4289194720,v1:172.21.15.123:6839/4289194720] up:active 2024-01-24T14:25:05.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:05 smithi123 ceph-mon[44966]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-01-24T14:25:05.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:05 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/1153400139' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-01-24T14:25:05.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:05 smithi123 ceph-mon[44966]: osdmap e78: 12 total, 12 up, 12 in 2024-01-24T14:25:05.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:05 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/1220630885' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-01-24T14:25:05.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:05 smithi194 ceph-mon[45450]: mds.? [v2:172.21.15.123:6838/4289194720,v1:172.21.15.123:6839/4289194720] up:active 2024-01-24T14:25:05.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:05 smithi194 ceph-mon[45450]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-01-24T14:25:05.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:05 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/1153400139' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-01-24T14:25:05.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:05 smithi194 ceph-mon[45450]: osdmap e78: 12 total, 12 up, 12 in 2024-01-24T14:25:05.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:05 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/1220630885' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-01-24T14:25:06.499 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.499+0000 7f67277fe700 1 -- 172.21.15.19:0/1220630885 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v80) v1 ==== 228+0+0 (secure 0 0 0) 0x7f6720026100 con 0x7f6730100e20 2024-01-24T14:25:06.504 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.504+0000 7f6738589700 1 -- 172.21.15.19:0/1220630885 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f671c0a0640 msgr2=0x7f671c0a2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:06.504 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.504+0000 7f6738589700 1 --2- 172.21.15.19:0/1220630885 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f671c0a0640 0x7f671c0a2b00 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f672c005950 tx=0x7f672c0058e0 comp rx=0 tx=0).stop 2024-01-24T14:25:06.504 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.504+0000 7f6738589700 1 -- 172.21.15.19:0/1220630885 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6730100e20 msgr2=0x7f6730199230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:06.504 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.504+0000 7f6738589700 1 --2- 172.21.15.19:0/1220630885 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6730100e20 0x7f6730199230 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f672000e9a0 tx=0x7f672000ed60 comp rx=0 tx=0).stop 2024-01-24T14:25:06.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.507+0000 7f6738589700 1 -- 172.21.15.19:0/1220630885 shutdown_connections 2024-01-24T14:25:06.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.507+0000 7f6738589700 1 --2- 172.21.15.19:0/1220630885 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f67301017c0 0x7f67301a0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:06.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.507+0000 7f6738589700 1 --2- 172.21.15.19:0/1220630885 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f67300ffc30 0x7f6730198cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:06.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.507+0000 7f6738589700 1 --2- 172.21.15.19:0/1220630885 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f671c0a0640 0x7f671c0a2b00 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:06.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.507+0000 7f6738589700 1 --2- 172.21.15.19:0/1220630885 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6730100e20 0x7f6730199230 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:06.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.507+0000 7f6738589700 1 -- 172.21.15.19:0/1220630885 >> 172.21.15.19:0/1220630885 conn(0x7f67300fb1a0 msgr2=0x7f6730067f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:06.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.508+0000 7f6738589700 1 -- 172.21.15.19:0/1220630885 shutdown_connections 2024-01-24T14:25:06.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.508+0000 7f6738589700 1 -- 172.21.15.19:0/1220630885 wait complete. 2024-01-24T14:25:06.511 INFO:teuthology.orchestra.run.smithi019.stderr:pool 'cephfs_data_ec' created 2024-01-24T14:25:06.525 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-01-24T14:25:06.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.669+0000 7f09127ee700 1 -- 172.21.15.19:0/903112369 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f090c1041b0 msgr2=0x7f090c1065a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:06.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.669+0000 7f09127ee700 1 --2- 172.21.15.19:0/903112369 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f090c1041b0 0x7f090c1065a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0900009990 tx=0x7f0900009ca0 comp rx=0 tx=0).stop 2024-01-24T14:25:06.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.669+0000 7f09127ee700 1 -- 172.21.15.19:0/903112369 shutdown_connections 2024-01-24T14:25:06.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.669+0000 7f09127ee700 1 --2- 172.21.15.19:0/903112369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f090c106ae0 0x7f090c108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:06.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.669+0000 7f09127ee700 1 --2- 172.21.15.19:0/903112369 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f090c101880 0x7f090c103c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:06.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.669+0000 7f09127ee700 1 --2- 172.21.15.19:0/903112369 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f090c1041b0 0x7f090c1065a0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:06.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.669+0000 7f09127ee700 1 -- 172.21.15.19:0/903112369 >> 172.21.15.19:0/903112369 conn(0x7f090c0fb210 msgr2=0x7f090c0fd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:06.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.669+0000 7f09127ee700 1 -- 172.21.15.19:0/903112369 shutdown_connections 2024-01-24T14:25:06.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.669+0000 7f09127ee700 1 -- 172.21.15.19:0/903112369 wait complete. 2024-01-24T14:25:06.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.670+0000 7f09127ee700 1 Processor -- start 2024-01-24T14:25:06.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.670+0000 7f09127ee700 1 -- start start 2024-01-24T14:25:06.672 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.670+0000 7f09127ee700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f090c101880 0x7f090c1011f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:06.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.670+0000 7f09127ee700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f090c1041b0 0x7f090c0ff830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:06.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.670+0000 7f09127ee700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f090c106ae0 0x7f090c0ffd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:06.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.670+0000 7f09127ee700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f090c100510 con 0x7f090c1041b0 2024-01-24T14:25:06.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.670+0000 7f09127ee700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f090c100680 con 0x7f090c101880 2024-01-24T14:25:06.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.670+0000 7f09127ee700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f090c1a0a40 con 0x7f090c106ae0 2024-01-24T14:25:06.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.670+0000 7f090b7fe700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f090c1041b0 0x7f090c0ff830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:06.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.670+0000 7f090b7fe700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f090c1041b0 0x7f090c0ff830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:32840/0 (socket says 172.21.15.19:32840) 2024-01-24T14:25:06.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.670+0000 7f090b7fe700 1 -- 172.21.15.19:0/2721565058 learned_addr learned my addr 172.21.15.19:0/2721565058 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:25:06.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.670+0000 7f090bfff700 1 --2- 172.21.15.19:0/2721565058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f090c101880 0x7f090c1011f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:06.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.671+0000 7f0910d8b700 1 --2- 172.21.15.19:0/2721565058 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f090c106ae0 0x7f090c0ffd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:06.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.671+0000 7f090b7fe700 1 -- 172.21.15.19:0/2721565058 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f090c106ae0 msgr2=0x7f090c0ffd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:06.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.671+0000 7f090b7fe700 1 --2- 172.21.15.19:0/2721565058 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f090c106ae0 0x7f090c0ffd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:06.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.671+0000 7f090b7fe700 1 -- 172.21.15.19:0/2721565058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f090c101880 msgr2=0x7f090c1011f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:06.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.671+0000 7f090b7fe700 1 --2- 172.21.15.19:0/2721565058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f090c101880 0x7f090c1011f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:06.674 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.671+0000 7f090b7fe700 1 -- 172.21.15.19:0/2721565058 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09000095f0 con 0x7f090c1041b0 2024-01-24T14:25:06.674 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.671+0000 7f090bfff700 1 --2- 172.21.15.19:0/2721565058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f090c101880 0x7f090c1011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T14:25:06.674 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.671+0000 7f090b7fe700 1 --2- 172.21.15.19:0/2721565058 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f090c1041b0 0x7f090c0ff830 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f09000059d0 tx=0x7f0900005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:06.674 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.671+0000 7f09097fa700 1 -- 172.21.15.19:0/2721565058 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f090001d070 con 0x7f090c1041b0 2024-01-24T14:25:06.674 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.671+0000 7f09097fa700 1 -- 172.21.15.19:0/2721565058 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f090000fc40 con 0x7f090c1041b0 2024-01-24T14:25:06.674 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.671+0000 7f09127ee700 1 -- 172.21.15.19:0/2721565058 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f090c1a0cc0 con 0x7f090c1041b0 2024-01-24T14:25:06.674 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.671+0000 7f09097fa700 1 -- 172.21.15.19:0/2721565058 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0900017650 con 0x7f090c1041b0 2024-01-24T14:25:06.674 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.671+0000 7f09127ee700 1 -- 172.21.15.19:0/2721565058 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f090c1a1140 con 0x7f090c1041b0 2024-01-24T14:25:06.674 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.672+0000 7f09127ee700 1 -- 172.21.15.19:0/2721565058 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f090c0fcd90 con 0x7f090c1041b0 2024-01-24T14:25:06.675 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.673+0000 7f09097fa700 1 -- 172.21.15.19:0/2721565058 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f0900022470 con 0x7f090c1041b0 2024-01-24T14:25:06.675 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.674+0000 7f09097fa700 1 --2- 172.21.15.19:0/2721565058 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f08f40a05a0 0x7f08f40a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:06.675 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.674+0000 7f09097fa700 1 -- 172.21.15.19:0/2721565058 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8698+0+0 (secure 0 0 0) 0x7f09000cd850 con 0x7f090c1041b0 2024-01-24T14:25:06.675 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.674+0000 7f090bfff700 1 --2- 172.21.15.19:0/2721565058 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f08f40a05a0 0x7f08f40a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:06.675 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.675+0000 7f090bfff700 1 --2- 172.21.15.19:0/2721565058 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f08f40a05a0 0x7f08f40a2a60 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f08fc009cb0 tx=0x7f08fc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:06.678 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.678+0000 7f09097fa700 1 -- 172.21.15.19:0/2721565058 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f09000972b0 con 0x7f090c1041b0 2024-01-24T14:25:06.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:06 smithi019 ceph-mon[40663]: pgmap v219: 97 pgs: 16 creating+peering, 21 active+clean, 60 unknown; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:25:06.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:06 smithi019 ceph-mon[40663]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-01-24T14:25:06.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:06 smithi019 ceph-mon[40663]: Cluster is now healthy 2024-01-24T14:25:06.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:06 smithi019 ceph-mon[40663]: osdmap e79: 12 total, 12 up, 12 in 2024-01-24T14:25:06.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:06 smithi123 ceph-mon[44966]: pgmap v219: 97 pgs: 16 creating+peering, 21 active+clean, 60 unknown; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:25:06.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:06 smithi123 ceph-mon[44966]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-01-24T14:25:06.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:06 smithi123 ceph-mon[44966]: Cluster is now healthy 2024-01-24T14:25:06.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:06 smithi123 ceph-mon[44966]: osdmap e79: 12 total, 12 up, 12 in 2024-01-24T14:25:06.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:06 smithi194 ceph-mon[45450]: pgmap v219: 97 pgs: 16 creating+peering, 21 active+clean, 60 unknown; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-01-24T14:25:06.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:06 smithi194 ceph-mon[45450]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-01-24T14:25:06.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:06 smithi194 ceph-mon[45450]: Cluster is now healthy 2024-01-24T14:25:06.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:06 smithi194 ceph-mon[45450]: osdmap e79: 12 total, 12 up, 12 in 2024-01-24T14:25:06.893 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:06.893+0000 7f09127ee700 1 -- 172.21.15.19:0/2721565058 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f090c02ce80 con 0x7f090c1041b0 2024-01-24T14:25:07.509 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.508+0000 7f09097fa700 1 -- 172.21.15.19:0/2721565058 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v81) v1 ==== 169+0+0 (secure 0 0 0) 0x7f090009a8d0 con 0x7f090c1041b0 2024-01-24T14:25:07.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.514+0000 7f09127ee700 1 -- 172.21.15.19:0/2721565058 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f08f40a05a0 msgr2=0x7f08f40a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:07.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.514+0000 7f09127ee700 1 --2- 172.21.15.19:0/2721565058 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f08f40a05a0 0x7f08f40a2a60 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f08fc009cb0 tx=0x7f08fc009260 comp rx=0 tx=0).stop 2024-01-24T14:25:07.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.514+0000 7f09127ee700 1 -- 172.21.15.19:0/2721565058 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f090c1041b0 msgr2=0x7f090c0ff830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:07.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.514+0000 7f09127ee700 1 --2- 172.21.15.19:0/2721565058 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f090c1041b0 0x7f090c0ff830 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f09000059d0 tx=0x7f0900005ef0 comp rx=0 tx=0).stop 2024-01-24T14:25:07.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.514+0000 7f09127ee700 1 -- 172.21.15.19:0/2721565058 shutdown_connections 2024-01-24T14:25:07.519 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.514+0000 7f09127ee700 1 --2- 172.21.15.19:0/2721565058 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f090c106ae0 0x7f090c0ffd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:07.519 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.514+0000 7f09127ee700 1 --2- 172.21.15.19:0/2721565058 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f090c101880 0x7f090c1011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:07.519 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.514+0000 7f09127ee700 1 --2- 172.21.15.19:0/2721565058 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f08f40a05a0 0x7f08f40a2a60 secure :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f08fc009cb0 tx=0x7f08fc009260 comp rx=0 tx=0).stop 2024-01-24T14:25:07.519 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.514+0000 7f09127ee700 1 --2- 172.21.15.19:0/2721565058 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f090c1041b0 0x7f090c0ff830 secure :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f09000059d0 tx=0x7f0900005ef0 comp rx=0 tx=0).stop 2024-01-24T14:25:07.519 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.514+0000 7f09127ee700 1 -- 172.21.15.19:0/2721565058 >> 172.21.15.19:0/2721565058 conn(0x7f090c0fb210 msgr2=0x7f090c1024f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:07.521 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.521+0000 7f09127ee700 1 -- 172.21.15.19:0/2721565058 shutdown_connections 2024-01-24T14:25:07.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.528+0000 7f09127ee700 1 -- 172.21.15.19:0/2721565058 wait complete. 2024-01-24T14:25:07.532 INFO:teuthology.orchestra.run.smithi019.stderr:set pool 4 allow_ec_overwrites to true 2024-01-24T14:25:07.546 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-01-24T14:25:07.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.687+0000 7fbf9dfc2700 1 -- 172.21.15.19:0/1148460131 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbf981017c0 msgr2=0x7fbf98103bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:07.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.687+0000 7fbf9dfc2700 1 --2- 172.21.15.19:0/1148460131 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbf981017c0 0x7fbf98103bb0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fbf88009990 tx=0x7fbf88009ca0 comp rx=0 tx=0).stop 2024-01-24T14:25:07.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.688+0000 7fbf9dfc2700 1 -- 172.21.15.19:0/1148460131 shutdown_connections 2024-01-24T14:25:07.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.688+0000 7fbf9dfc2700 1 --2- 172.21.15.19:0/1148460131 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbf981017c0 0x7fbf98103bb0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:07.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.688+0000 7fbf9dfc2700 1 --2- 172.21.15.19:0/1148460131 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbf98106a20 0x7fbf98108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:07.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.688+0000 7fbf9dfc2700 1 --2- 172.21.15.19:0/1148460131 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf981040f0 0x7fbf981064e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:07.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.688+0000 7fbf9dfc2700 1 -- 172.21.15.19:0/1148460131 >> 172.21.15.19:0/1148460131 conn(0x7fbf980fb150 msgr2=0x7fbf980fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:07.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.688+0000 7fbf9dfc2700 1 -- 172.21.15.19:0/1148460131 shutdown_connections 2024-01-24T14:25:07.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.688+0000 7fbf9dfc2700 1 -- 172.21.15.19:0/1148460131 wait complete. 2024-01-24T14:25:07.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.688+0000 7fbf9dfc2700 1 Processor -- start 2024-01-24T14:25:07.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.688+0000 7fbf9dfc2700 1 -- start start 2024-01-24T14:25:07.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.689+0000 7fbf9dfc2700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbf981017c0 0x7fbf98198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:07.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.689+0000 7fbf9dfc2700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf981040f0 0x7fbf98199160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:07.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.689+0000 7fbf9dfc2700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbf98106a20 0x7fbf981a0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:07.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.689+0000 7fbf9dfc2700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf981a0760 con 0x7fbf98106a20 2024-01-24T14:25:07.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.689+0000 7fbf9dfc2700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf981a08d0 con 0x7fbf981040f0 2024-01-24T14:25:07.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.689+0000 7fbf9dfc2700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf981a0a90 con 0x7fbf981017c0 2024-01-24T14:25:07.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.689+0000 7fbf97fff700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbf98106a20 0x7fbf981a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:07.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.689+0000 7fbf97fff700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbf98106a20 0x7fbf981a0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:32856/0 (socket says 172.21.15.19:32856) 2024-01-24T14:25:07.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.689+0000 7fbf97fff700 1 -- 172.21.15.19:0/3591851634 learned_addr learned my addr 172.21.15.19:0/3591851634 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:25:07.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.689+0000 7fbf96ffd700 1 --2- 172.21.15.19:0/3591851634 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf981040f0 0x7fbf98199160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:07.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.689+0000 7fbf977fe700 1 --2- 172.21.15.19:0/3591851634 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbf981017c0 0x7fbf98198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:07.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.689+0000 7fbf97fff700 1 -- 172.21.15.19:0/3591851634 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbf981017c0 msgr2=0x7fbf98198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:07.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.689+0000 7fbf97fff700 1 --2- 172.21.15.19:0/3591851634 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbf981017c0 0x7fbf98198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:07.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.689+0000 7fbf97fff700 1 -- 172.21.15.19:0/3591851634 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf981040f0 msgr2=0x7fbf98199160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:07.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.689+0000 7fbf97fff700 1 --2- 172.21.15.19:0/3591851634 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf981040f0 0x7fbf98199160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:07.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.689+0000 7fbf97fff700 1 -- 172.21.15.19:0/3591851634 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf880095f0 con 0x7fbf98106a20 2024-01-24T14:25:07.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.689+0000 7fbf97fff700 1 --2- 172.21.15.19:0/3591851634 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbf98106a20 0x7fbf981a0190 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fbf90007ee0 tx=0x7fbf9000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:07.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.689+0000 7fbf94ff9700 1 -- 172.21.15.19:0/3591851634 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf90017070 con 0x7fbf98106a20 2024-01-24T14:25:07.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.689+0000 7fbf94ff9700 1 -- 172.21.15.19:0/3591851634 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbf9000f040 con 0x7fbf98106a20 2024-01-24T14:25:07.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.689+0000 7fbf9dfc2700 1 -- 172.21.15.19:0/3591851634 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf981a0d70 con 0x7fbf98106a20 2024-01-24T14:25:07.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.689+0000 7fbf94ff9700 1 -- 172.21.15.19:0/3591851634 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf90005580 con 0x7fbf98106a20 2024-01-24T14:25:07.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.690+0000 7fbf9dfc2700 1 -- 172.21.15.19:0/3591851634 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf981a11d0 con 0x7fbf98106a20 2024-01-24T14:25:07.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.691+0000 7fbf9dfc2700 1 -- 172.21.15.19:0/3591851634 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf981085a0 con 0x7fbf98106a20 2024-01-24T14:25:07.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.692+0000 7fbf94ff9700 1 -- 172.21.15.19:0/3591851634 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7fbf9000db20 con 0x7fbf98106a20 2024-01-24T14:25:07.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.693+0000 7fbf94ff9700 1 --2- 172.21.15.19:0/3591851634 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fbf800a05a0 0x7fbf800a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:07.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.693+0000 7fbf977fe700 1 --2- 172.21.15.19:0/3591851634 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fbf800a05a0 0x7fbf800a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:07.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.693+0000 7fbf94ff9700 1 -- 172.21.15.19:0/3591851634 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8698+0+0 (secure 0 0 0) 0x7fbf900cb880 con 0x7fbf98106a20 2024-01-24T14:25:07.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.693+0000 7fbf977fe700 1 --2- 172.21.15.19:0/3591851634 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fbf800a05a0 0x7fbf800a2a60 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbf8800b5c0 tx=0x7fbf88005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:07.696 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.696+0000 7fbf94ff9700 1 -- 172.21.15.19:0/3591851634 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fbf90095250 con 0x7fbf98106a20 2024-01-24T14:25:07.799 INFO:journalctl@ceph.mds.b.smithi123.stdout:Jan 24 14:25:07 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-b[81360]: 2024-01-24T14:25:07.458+0000 7f37f6c96700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-24T14:25:07.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:07 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/1220630885' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-01-24T14:25:07.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:07 smithi123 ceph-mon[44966]: osdmap e80: 12 total, 12 up, 12 in 2024-01-24T14:25:07.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:07 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/2721565058' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-01-24T14:25:07.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:07 smithi123 ceph-mon[44966]: pgmap v222: 129 pgs: 16 creating+peering, 64 active+clean, 49 unknown; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 777 B/s wr, 2 op/s 2024-01-24T14:25:07.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:07 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/1220630885' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-01-24T14:25:07.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:07 smithi194 ceph-mon[45450]: osdmap e80: 12 total, 12 up, 12 in 2024-01-24T14:25:07.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:07 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/2721565058' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-01-24T14:25:07.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:07 smithi194 ceph-mon[45450]: pgmap v222: 129 pgs: 16 creating+peering, 64 active+clean, 49 unknown; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 777 B/s wr, 2 op/s 2024-01-24T14:25:07.960 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.960+0000 7fbf9dfc2700 1 -- 172.21.15.19:0/3591851634 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7fbf9802cc70 con 0x7fbf98106a20 2024-01-24T14:25:07.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.967+0000 7fbf94ff9700 1 -- 172.21.15.19:0/3591851634 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v16) v1 ==== 137+0+0 (secure 0 0 0) 0x7fbf90098870 con 0x7fbf98106a20 2024-01-24T14:25:07.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.970+0000 7fbf9dfc2700 1 -- 172.21.15.19:0/3591851634 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fbf800a05a0 msgr2=0x7fbf800a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:07.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.970+0000 7fbf9dfc2700 1 --2- 172.21.15.19:0/3591851634 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fbf800a05a0 0x7fbf800a2a60 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbf8800b5c0 tx=0x7fbf88005fb0 comp rx=0 tx=0).stop 2024-01-24T14:25:07.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.970+0000 7fbf9dfc2700 1 -- 172.21.15.19:0/3591851634 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbf98106a20 msgr2=0x7fbf981a0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:07.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.970+0000 7fbf9dfc2700 1 --2- 172.21.15.19:0/3591851634 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbf98106a20 0x7fbf981a0190 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fbf90007ee0 tx=0x7fbf9000d3b0 comp rx=0 tx=0).stop 2024-01-24T14:25:07.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.970+0000 7fbf9dfc2700 1 -- 172.21.15.19:0/3591851634 shutdown_connections 2024-01-24T14:25:07.971 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.970+0000 7fbf9dfc2700 1 --2- 172.21.15.19:0/3591851634 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fbf800a05a0 0x7fbf800a2a60 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:07.971 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.970+0000 7fbf9dfc2700 1 --2- 172.21.15.19:0/3591851634 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbf98106a20 0x7fbf981a0190 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:07.971 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.970+0000 7fbf9dfc2700 1 --2- 172.21.15.19:0/3591851634 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbf981017c0 0x7fbf98198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:07.971 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.970+0000 7fbf9dfc2700 1 --2- 172.21.15.19:0/3591851634 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf981040f0 0x7fbf98199160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:07.971 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.970+0000 7fbf9dfc2700 1 -- 172.21.15.19:0/3591851634 >> 172.21.15.19:0/3591851634 conn(0x7fbf980fb150 msgr2=0x7fbf980ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:07.971 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.970+0000 7fbf9dfc2700 1 -- 172.21.15.19:0/3591851634 shutdown_connections 2024-01-24T14:25:07.971 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:07.970+0000 7fbf9dfc2700 1 -- 172.21.15.19:0/3591851634 wait complete. 2024-01-24T14:25:07.971 INFO:teuthology.orchestra.run.smithi019.stderr:added data pool 4 to fsmap 2024-01-24T14:25:07.983 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:07 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/1220630885' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-01-24T14:25:07.983 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:07 smithi019 ceph-mon[40663]: osdmap e80: 12 total, 12 up, 12 in 2024-01-24T14:25:07.983 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:07 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/2721565058' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-01-24T14:25:07.983 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:07 smithi019 ceph-mon[40663]: pgmap v222: 129 pgs: 16 creating+peering, 64 active+clean, 49 unknown; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 777 B/s wr, 2 op/s 2024-01-24T14:25:07.984 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T14:25:08.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.122+0000 7f1162e25700 1 -- 172.21.15.19:0/2187593536 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f115c1017b0 msgr2=0x7f115c108060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:08.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.122+0000 7f1162e25700 1 --2- 172.21.15.19:0/2187593536 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f115c1017b0 0x7f115c108060 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f115400b0d0 tx=0x7f115400b3e0 comp rx=0 tx=0).stop 2024-01-24T14:25:08.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.122+0000 7f1162e25700 1 -- 172.21.15.19:0/2187593536 shutdown_connections 2024-01-24T14:25:08.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.122+0000 7f1162e25700 1 --2- 172.21.15.19:0/2187593536 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f115c1017b0 0x7f115c108060 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:08.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.122+0000 7f1162e25700 1 --2- 172.21.15.19:0/2187593536 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f115c0ffc20 0x7f115c100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:08.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.122+0000 7f1162e25700 1 --2- 172.21.15.19:0/2187593536 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f115c100e10 0x7f115c101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:08.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.122+0000 7f1162e25700 1 -- 172.21.15.19:0/2187593536 >> 172.21.15.19:0/2187593536 conn(0x7f115c0fb170 msgr2=0x7f115c0fd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:08.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.122+0000 7f1162e25700 1 -- 172.21.15.19:0/2187593536 shutdown_connections 2024-01-24T14:25:08.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.122+0000 7f1162e25700 1 -- 172.21.15.19:0/2187593536 wait complete. 2024-01-24T14:25:08.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.122+0000 7f1162e25700 1 Processor -- start 2024-01-24T14:25:08.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.122+0000 7f1162e25700 1 -- start start 2024-01-24T14:25:08.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.123+0000 7f1162e25700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f115c0ffc20 0x7f115c198cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:08.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.123+0000 7f1162e25700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f115c100e10 0x7f115c199200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:08.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.123+0000 7f1162e25700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f115c1017b0 0x7f115c1a0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:08.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.123+0000 7f1162e25700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f115c1a0830 con 0x7f115c100e10 2024-01-24T14:25:08.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.123+0000 7f1162e25700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f115c1a09a0 con 0x7f115c1017b0 2024-01-24T14:25:08.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.123+0000 7f1162e25700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f115c1a0b60 con 0x7f115c0ffc20 2024-01-24T14:25:08.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.123+0000 7f115bfff700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f115c100e10 0x7f115c199200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:08.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.123+0000 7f115bfff700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f115c100e10 0x7f115c199200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:32870/0 (socket says 172.21.15.19:32870) 2024-01-24T14:25:08.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.123+0000 7f115bfff700 1 -- 172.21.15.19:0/2925370749 learned_addr learned my addr 172.21.15.19:0/2925370749 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:25:08.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.123+0000 7f1160bc1700 1 --2- 172.21.15.19:0/2925370749 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f115c0ffc20 0x7f115c198cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:08.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.123+0000 7f11613c2700 1 --2- 172.21.15.19:0/2925370749 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f115c1017b0 0x7f115c1a0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:08.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.123+0000 7f115bfff700 1 -- 172.21.15.19:0/2925370749 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f115c0ffc20 msgr2=0x7f115c198cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:08.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.123+0000 7f115bfff700 1 --2- 172.21.15.19:0/2925370749 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f115c0ffc20 0x7f115c198cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:08.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.123+0000 7f115bfff700 1 -- 172.21.15.19:0/2925370749 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f115c1017b0 msgr2=0x7f115c1a0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:08.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.123+0000 7f115bfff700 1 --2- 172.21.15.19:0/2925370749 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f115c1017b0 0x7f115c1a0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:08.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.123+0000 7f115bfff700 1 -- 172.21.15.19:0/2925370749 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f114c009520 con 0x7f115c100e10 2024-01-24T14:25:08.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.123+0000 7f11613c2700 1 --2- 172.21.15.19:0/2925370749 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f115c1017b0 0x7f115c1a0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T14:25:08.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.123+0000 7f115bfff700 1 --2- 172.21.15.19:0/2925370749 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f115c100e10 0x7f115c199200 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f114400e9a0 tx=0x7f114400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:08.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.123+0000 7f1159ffb700 1 -- 172.21.15.19:0/2925370749 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1144009790 con 0x7f115c100e10 2024-01-24T14:25:08.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.123+0000 7f1159ffb700 1 -- 172.21.15.19:0/2925370749 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f114400cbd0 con 0x7f115c100e10 2024-01-24T14:25:08.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.123+0000 7f1162e25700 1 -- 172.21.15.19:0/2925370749 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1154009d00 con 0x7f115c100e10 2024-01-24T14:25:08.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.123+0000 7f1159ffb700 1 -- 172.21.15.19:0/2925370749 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1144018680 con 0x7f115c100e10 2024-01-24T14:25:08.128 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.124+0000 7f1162e25700 1 -- 172.21.15.19:0/2925370749 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f115c1a11c0 con 0x7f115c100e10 2024-01-24T14:25:08.129 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.125+0000 7f1162e25700 1 -- 172.21.15.19:0/2925370749 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f115c10b490 con 0x7f115c100e10 2024-01-24T14:25:08.129 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.126+0000 7f1159ffb700 1 -- 172.21.15.19:0/2925370749 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f114400cd40 con 0x7f115c100e10 2024-01-24T14:25:08.129 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.127+0000 7f1159ffb700 1 --2- 172.21.15.19:0/2925370749 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f11480a05f0 0x7f11480a2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:08.129 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.127+0000 7f1160bc1700 1 --2- 172.21.15.19:0/2925370749 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f11480a05f0 0x7f11480a2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:08.129 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.127+0000 7f1159ffb700 1 -- 172.21.15.19:0/2925370749 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f1144014070 con 0x7f115c100e10 2024-01-24T14:25:08.129 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.128+0000 7f1160bc1700 1 --2- 172.21.15.19:0/2925370749 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f11480a05f0 0x7f11480a2ab0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f115c199b90 tx=0x7f114c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:08.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.131+0000 7f1159ffb700 1 -- 172.21.15.19:0/2925370749 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f11440959f0 con 0x7f115c100e10 2024-01-24T14:25:08.396 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.396+0000 7f1162e25700 1 -- 172.21.15.19:0/2925370749 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f115c10b680 con 0x7f115c100e10 2024-01-24T14:25:08.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.397+0000 7f1159ffb700 1 -- 172.21.15.19:0/2925370749 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+10813 (secure 0 0 0) 0x7f11440b0a50 con 0x7f115c100e10 2024-01-24T14:25:08.397 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:25:08.397 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6835/1060583512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6834","nonce":1060583512},{"type":"v1","addr":"172.21.15.19:6835","nonce":1060583512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6837/2302964875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6836","nonce":2302964875},{"type":"v1","addr":"172.21.15.19:6837","nonce":2302964875}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14607,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6839/3376327575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6838","nonce":3376327575},{"type":"v1","addr":"172.21.15.19:6839","nonce":3376327575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14619,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6841/1702088084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6840","nonce":1702088084},{"type":"v1","addr":"172.21.15.19:6841","nonce":1702088084}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2647625404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2647625404},{"type":"v1","addr":"172.21.15.194:6833","nonce":2647625404}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/3396650565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3396650565},{"type":"v1","addr":"172.21.15.123:6833","nonce":3396650565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24485,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2631676083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2631676083},{"type":"v1","addr":"172.21.15.123:6835","nonce":2631676083}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6835/456696823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":456696823},{"type":"v1","addr":"172.21.15.194:6835","nonce":456696823}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6837/2623846234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":2623846234},{"type":"v1","addr":"172.21.15.194:6837","nonce":2623846234}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/491309336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":491309336},{"type":"v1","addr":"172.21.15.123:6837","nonce":491309336}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24514,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6839/810249223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":810249223},{"type":"v1","addr":"172.21.15.194:6839","nonce":810249223}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T14:25:03.484235+0000","modified":"2024-01-24T14:25:07.961163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24545},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6839/4289194720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":4289194720},{"type":"v1","addr":"172.21.15.123:6839","nonce":4289194720}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-01-24T14:25:08.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.399+0000 7f1162e25700 1 -- 172.21.15.19:0/2925370749 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f11480a05f0 msgr2=0x7f11480a2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:08.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.399+0000 7f1162e25700 1 --2- 172.21.15.19:0/2925370749 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f11480a05f0 0x7f11480a2ab0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f115c199b90 tx=0x7f114c009260 comp rx=0 tx=0).stop 2024-01-24T14:25:08.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.399+0000 7f1162e25700 1 -- 172.21.15.19:0/2925370749 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f115c100e10 msgr2=0x7f115c199200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:08.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.399+0000 7f1162e25700 1 --2- 172.21.15.19:0/2925370749 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f115c100e10 0x7f115c199200 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f114400e9a0 tx=0x7f114400ecb0 comp rx=0 tx=0).stop 2024-01-24T14:25:08.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.399+0000 7f1162e25700 1 -- 172.21.15.19:0/2925370749 shutdown_connections 2024-01-24T14:25:08.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.399+0000 7f1162e25700 1 --2- 172.21.15.19:0/2925370749 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f115c1017b0 0x7f115c1a0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:08.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.399+0000 7f1162e25700 1 --2- 172.21.15.19:0/2925370749 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f115c0ffc20 0x7f115c198cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:08.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.399+0000 7f1162e25700 1 --2- 172.21.15.19:0/2925370749 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f11480a05f0 0x7f11480a2ab0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:08.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.399+0000 7f1162e25700 1 --2- 172.21.15.19:0/2925370749 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f115c100e10 0x7f115c199200 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:08.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.399+0000 7f1162e25700 1 -- 172.21.15.19:0/2925370749 >> 172.21.15.19:0/2925370749 conn(0x7f115c0fb170 msgr2=0x7f115c067ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:08.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.399+0000 7f1162e25700 1 -- 172.21.15.19:0/2925370749 shutdown_connections 2024-01-24T14:25:08.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.400+0000 7f1162e25700 1 -- 172.21.15.19:0/2925370749 wait complete. 2024-01-24T14:25:08.401 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 16 2024-01-24T14:25:08.413 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T14:25:08.555 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.550+0000 7f3d323db700 1 -- 172.21.15.19:0/2769537252 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3d2c106a60 msgr2=0x7f3d2c108e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:08.555 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.550+0000 7f3d323db700 1 --2- 172.21.15.19:0/2769537252 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3d2c106a60 0x7f3d2c108e70 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3d2400b0a0 tx=0x7f3d2400b3b0 comp rx=0 tx=0).stop 2024-01-24T14:25:08.555 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.551+0000 7f3d323db700 1 -- 172.21.15.19:0/2769537252 shutdown_connections 2024-01-24T14:25:08.555 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.551+0000 7f3d323db700 1 --2- 172.21.15.19:0/2769537252 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3d2c106a60 0x7f3d2c108e70 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:08.555 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.551+0000 7f3d323db700 1 --2- 172.21.15.19:0/2769537252 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d2c101800 0x7f3d2c103bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:08.555 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.551+0000 7f3d323db700 1 --2- 172.21.15.19:0/2769537252 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3d2c104130 0x7f3d2c106520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:08.556 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.551+0000 7f3d323db700 1 -- 172.21.15.19:0/2769537252 >> 172.21.15.19:0/2769537252 conn(0x7f3d2c0fb170 msgr2=0x7f3d2c0fd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:08.556 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.551+0000 7f3d323db700 1 -- 172.21.15.19:0/2769537252 shutdown_connections 2024-01-24T14:25:08.556 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.551+0000 7f3d323db700 1 -- 172.21.15.19:0/2769537252 wait complete. 2024-01-24T14:25:08.556 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.551+0000 7f3d323db700 1 Processor -- start 2024-01-24T14:25:08.556 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.551+0000 7f3d323db700 1 -- start start 2024-01-24T14:25:08.556 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.552+0000 7f3d323db700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d2c101800 0x7f3d2c196b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:08.556 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.552+0000 7f3d323db700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3d2c104130 0x7f3d2c197090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:08.556 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.552+0000 7f3d323db700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3d2c106a60 0x7f3d2c19e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:08.557 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.552+0000 7f3d323db700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2c19e680 con 0x7f3d2c104130 2024-01-24T14:25:08.557 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.552+0000 7f3d323db700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2c19e7c0 con 0x7f3d2c101800 2024-01-24T14:25:08.557 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.552+0000 7f3d323db700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2c19e950 con 0x7f3d2c106a60 2024-01-24T14:25:08.557 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.552+0000 7f3d2b7fe700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3d2c104130 0x7f3d2c197090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:08.557 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.552+0000 7f3d2b7fe700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3d2c104130 0x7f3d2c197090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:32892/0 (socket says 172.21.15.19:32892) 2024-01-24T14:25:08.557 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.552+0000 7f3d2b7fe700 1 -- 172.21.15.19:0/680641486 learned_addr learned my addr 172.21.15.19:0/680641486 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:25:08.557 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.552+0000 7f3d30978700 1 --2- 172.21.15.19:0/680641486 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3d2c106a60 0x7f3d2c19e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:08.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.552+0000 7f3d2bfff700 1 --2- 172.21.15.19:0/680641486 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d2c101800 0x7f3d2c196b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:08.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.552+0000 7f3d2b7fe700 1 -- 172.21.15.19:0/680641486 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3d2c106a60 msgr2=0x7f3d2c19e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:08.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.552+0000 7f3d2b7fe700 1 --2- 172.21.15.19:0/680641486 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3d2c106a60 0x7f3d2c19e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:08.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.552+0000 7f3d2b7fe700 1 -- 172.21.15.19:0/680641486 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d2c101800 msgr2=0x7f3d2c196b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:08.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.552+0000 7f3d2b7fe700 1 --2- 172.21.15.19:0/680641486 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d2c101800 0x7f3d2c196b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:08.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.552+0000 7f3d2b7fe700 1 -- 172.21.15.19:0/680641486 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d24009d00 con 0x7f3d2c104130 2024-01-24T14:25:08.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.552+0000 7f3d2bfff700 1 --2- 172.21.15.19:0/680641486 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d2c101800 0x7f3d2c196b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T14:25:08.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.552+0000 7f3d2b7fe700 1 --2- 172.21.15.19:0/680641486 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3d2c104130 0x7f3d2c197090 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3d2000d980 tx=0x7f3d2000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:08.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.553+0000 7f3d297fa700 1 -- 172.21.15.19:0/680641486 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d20004220 con 0x7f3d2c104130 2024-01-24T14:25:08.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.553+0000 7f3d297fa700 1 -- 172.21.15.19:0/680641486 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d20007640 con 0x7f3d2c104130 2024-01-24T14:25:08.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.553+0000 7f3d323db700 1 -- 172.21.15.19:0/680641486 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d2c19ec30 con 0x7f3d2c104130 2024-01-24T14:25:08.559 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.553+0000 7f3d297fa700 1 -- 172.21.15.19:0/680641486 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d20005610 con 0x7f3d2c104130 2024-01-24T14:25:08.559 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.553+0000 7f3d323db700 1 -- 172.21.15.19:0/680641486 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d2c19f140 con 0x7f3d2c104130 2024-01-24T14:25:08.559 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.554+0000 7f3d323db700 1 -- 172.21.15.19:0/680641486 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d2c0fccf0 con 0x7f3d2c104130 2024-01-24T14:25:08.559 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.556+0000 7f3d297fa700 1 -- 172.21.15.19:0/680641486 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f3d20004750 con 0x7f3d2c104130 2024-01-24T14:25:08.559 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.556+0000 7f3d297fa700 1 --2- 172.21.15.19:0/680641486 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f3d140a0570 0x7f3d140a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:08.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.556+0000 7f3d2bfff700 1 --2- 172.21.15.19:0/680641486 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f3d140a0570 0x7f3d140a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:08.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.556+0000 7f3d297fa700 1 -- 172.21.15.19:0/680641486 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f3d200cbe60 con 0x7f3d2c104130 2024-01-24T14:25:08.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.557+0000 7f3d2bfff700 1 --2- 172.21.15.19:0/680641486 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f3d140a0570 0x7f3d140a2a30 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c009cb0 tx=0x7f3d1c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:08.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.561+0000 7f3d297fa700 1 -- 172.21.15.19:0/680641486 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f3d20095810 con 0x7f3d2c104130 2024-01-24T14:25:08.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:08 smithi019 ceph-mon[40663]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-01-24T14:25:08.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:08 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/2721565058' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-01-24T14:25:08.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:08 smithi019 ceph-mon[40663]: osdmap e81: 12 total, 12 up, 12 in 2024-01-24T14:25:08.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:08 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/3591851634' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-01-24T14:25:08.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:08 smithi019 ceph-mon[40663]: osdmap e82: 12 total, 12 up, 12 in 2024-01-24T14:25:08.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:08 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/3591851634' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-01-24T14:25:08.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:08 smithi019 ceph-mon[40663]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-01-24T14:25:08.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:08 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/2925370749' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T14:25:08.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.774+0000 7f3d323db700 1 -- 172.21.15.19:0/680641486 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d2c02cec0 con 0x7f3d2c104130 2024-01-24T14:25:08.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.775+0000 7f3d297fa700 1 -- 172.21.15.19:0/680641486 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+23997 (secure 0 0 0) 0x7f3d20098e30 con 0x7f3d2c104130 2024-01-24T14:25:08.776 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:25:08.776 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":82,"fsid":"654b0c8e-bac3-11ee-95b1-87774f69a715","created":"2024-01-24T14:19:10.699369+0000","modified":"2024-01-24T14:25:07.960902+0000","last_up_change":"2024-01-24T14:24:30.032711+0000","last_in_change":"2024-01-24T14:24:13.776095+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T14:21:53.453429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-24T14:25:01.967538+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-24T14:25:02.864396+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-01-24T14:25:05.950591+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"81380ec6-8e19-4eb4-8158-8fafaed8d7e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":2456786579},{"type":"v1","addr":"172.21.15.19:6803","nonce":2456786579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":2456786579},{"type":"v1","addr":"172.21.15.19:6805","nonce":2456786579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":2456786579},{"type":"v1","addr":"172.21.15.19:6809","nonce":2456786579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":2456786579},{"type":"v1","addr":"172.21.15.19:6807","nonce":2456786579}]},"public_addr":"172.21.15.19:6803/2456786579","cluster_addr":"172.21.15.19:6805/2456786579","heartbeat_back_addr":"172.21.15.19:6809/2456786579","heartbeat_front_addr":"172.21.15.19:6807/2456786579","state":["exists","up"]},{"osd":1,"uuid":"687f5036-a6d2-4aa1-86e7-7570e51ef179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":2407548889},{"type":"v1","addr":"172.21.15.123:6801","nonce":2407548889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":2407548889},{"type":"v1","addr":"172.21.15.123:6803","nonce":2407548889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":2407548889},{"type":"v1","addr":"172.21.15.123:6807","nonce":2407548889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":2407548889},{"type":"v1","addr":"172.21.15.123:6805","nonce":2407548889}]},"public_addr":"172.21.15.123:6801/2407548889","cluster_addr":"172.21.15.123:6803/2407548889","heartbeat_back_addr":"172.21.15.123:6807/2407548889","heartbeat_front_addr":"172.21.15.123:6805/2407548889","state":["exists","up"]},{"osd":2,"uuid":"3e810904-371d-4ca8-b43c-434957a2aa12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":3632698637},{"type":"v1","addr":"172.21.15.194:6801","nonce":3632698637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":3632698637},{"type":"v1","addr":"172.21.15.194:6803","nonce":3632698637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":3632698637},{"type":"v1","addr":"172.21.15.194:6807","nonce":3632698637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":3632698637},{"type":"v1","addr":"172.21.15.194:6805","nonce":3632698637}]},"public_addr":"172.21.15.194:6801/3632698637","cluster_addr":"172.21.15.194:6803/3632698637","heartbeat_back_addr":"172.21.15.194:6807/3632698637","heartbeat_front_addr":"172.21.15.194:6805/3632698637","state":["exists","up"]},{"osd":3,"uuid":"3665d6e0-010a-40bd-b0c6-682af56a0cbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":3868219159},{"type":"v1","addr":"172.21.15.19:6811","nonce":3868219159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3868219159},{"type":"v1","addr":"172.21.15.19:6813","nonce":3868219159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":3868219159},{"type":"v1","addr":"172.21.15.19:6817","nonce":3868219159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":3868219159},{"type":"v1","addr":"172.21.15.19:6815","nonce":3868219159}]},"public_addr":"172.21.15.19:6811/3868219159","cluster_addr":"172.21.15.19:6813/3868219159","heartbeat_back_addr":"172.21.15.19:6817/3868219159","heartbeat_front_addr":"172.21.15.19:6815/3868219159","state":["exists","up"]},{"osd":4,"uuid":"029fb28f-9af5-4239-8ee5-5ac5d04cd34b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":3611579809},{"type":"v1","addr":"172.21.15.123:6809","nonce":3611579809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":3611579809},{"type":"v1","addr":"172.21.15.123:6811","nonce":3611579809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":3611579809},{"type":"v1","addr":"172.21.15.123:6815","nonce":3611579809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":3611579809},{"type":"v1","addr":"172.21.15.123:6813","nonce":3611579809}]},"public_addr":"172.21.15.123:6809/3611579809","cluster_addr":"172.21.15.123:6811/3611579809","heartbeat_back_addr":"172.21.15.123:6815/3611579809","heartbeat_front_addr":"172.21.15.123:6813/3611579809","state":["exists","up"]},{"osd":5,"uuid":"6e97f23b-f803-4fed-a0a2-7aee7de4970f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":2483361871},{"type":"v1","addr":"172.21.15.194:6809","nonce":2483361871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":2483361871},{"type":"v1","addr":"172.21.15.194:6811","nonce":2483361871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":2483361871},{"type":"v1","addr":"172.21.15.194:6815","nonce":2483361871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":2483361871},{"type":"v1","addr":"172.21.15.194:6813","nonce":2483361871}]},"public_addr":"172.21.15.194:6809/2483361871","cluster_addr":"172.21.15.194:6811/2483361871","heartbeat_back_addr":"172.21.15.194:6815/2483361871","heartbeat_front_addr":"172.21.15.194:6813/2483361871","state":["exists","up"]},{"osd":6,"uuid":"c8c8ca2e-4cc6-4861-94b0-6211e4936c25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":1290513386},{"type":"v1","addr":"172.21.15.19:6819","nonce":1290513386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":1290513386},{"type":"v1","addr":"172.21.15.19:6821","nonce":1290513386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":1290513386},{"type":"v1","addr":"172.21.15.19:6825","nonce":1290513386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":1290513386},{"type":"v1","addr":"172.21.15.19:6823","nonce":1290513386}]},"public_addr":"172.21.15.19:6819/1290513386","cluster_addr":"172.21.15.19:6821/1290513386","heartbeat_back_addr":"172.21.15.19:6825/1290513386","heartbeat_front_addr":"172.21.15.19:6823/1290513386","state":["exists","up"]},{"osd":7,"uuid":"9aaea7a8-7bc3-4e32-9328-20a6c30a6d8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1391966446},{"type":"v1","addr":"172.21.15.123:6817","nonce":1391966446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1391966446},{"type":"v1","addr":"172.21.15.123:6819","nonce":1391966446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1391966446},{"type":"v1","addr":"172.21.15.123:6823","nonce":1391966446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1391966446},{"type":"v1","addr":"172.21.15.123:6821","nonce":1391966446}]},"public_addr":"172.21.15.123:6817/1391966446","cluster_addr":"172.21.15.123:6819/1391966446","heartbeat_back_addr":"172.21.15.123:6823/1391966446","heartbeat_front_addr":"172.21.15.123:6821/1391966446","state":["exists","up"]},{"osd":8,"uuid":"e918ec1b-7608-4f73-9067-29082ea63999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":2260577139},{"type":"v1","addr":"172.21.15.194:6817","nonce":2260577139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":2260577139},{"type":"v1","addr":"172.21.15.194:6819","nonce":2260577139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":2260577139},{"type":"v1","addr":"172.21.15.194:6823","nonce":2260577139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":2260577139},{"type":"v1","addr":"172.21.15.194:6821","nonce":2260577139}]},"public_addr":"172.21.15.194:6817/2260577139","cluster_addr":"172.21.15.194:6819/2260577139","heartbeat_back_addr":"172.21.15.194:6823/2260577139","heartbeat_front_addr":"172.21.15.194:6821/2260577139","state":["exists","up"]},{"osd":9,"uuid":"0fd76722-3319-4828-ab68-108607a4bd0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":4004843075},{"type":"v1","addr":"172.21.15.19:6827","nonce":4004843075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":4004843075},{"type":"v1","addr":"172.21.15.19:6829","nonce":4004843075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":4004843075},{"type":"v1","addr":"172.21.15.19:6833","nonce":4004843075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":4004843075},{"type":"v1","addr":"172.21.15.19:6831","nonce":4004843075}]},"public_addr":"172.21.15.19:6827/4004843075","cluster_addr":"172.21.15.19:6829/4004843075","heartbeat_back_addr":"172.21.15.19:6833/4004843075","heartbeat_front_addr":"172.21.15.19:6831/4004843075","state":["exists","up"]},{"osd":10,"uuid":"624bb8f5-6d72-43d5-9986-1b48b8a564ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2591845068},{"type":"v1","addr":"172.21.15.123:6825","nonce":2591845068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":2591845068},{"type":"v1","addr":"172.21.15.123:6827","nonce":2591845068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2591845068},{"type":"v1","addr":"172.21.15.123:6831","nonce":2591845068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2591845068},{"type":"v1","addr":"172.21.15.123:6829","nonce":2591845068}]},"public_addr":"172.21.15.123:6825/2591845068","cluster_addr":"172.21.15.123:6827/2591845068","heartbeat_back_addr":"172.21.15.123:6831/2591845068","heartbeat_front_addr":"172.21.15.123:6829/2591845068","state":["exists","up"]},{"osd":11,"uuid":"219a55b3-b01e-471d-90da-1a3043bf3e34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":3985496260},{"type":"v1","addr":"172.21.15.194:6825","nonce":3985496260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":3985496260},{"type":"v1","addr":"172.21.15.194:6827","nonce":3985496260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":3985496260},{"type":"v1","addr":"172.21.15.194:6831","nonce":3985496260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":3985496260},{"type":"v1","addr":"172.21.15.194:6829","nonce":3985496260}]},"public_addr":"172.21.15.194:6825/3985496260","cluster_addr":"172.21.15.194:6827/3985496260","heartbeat_back_addr":"172.21.15.194:6831/3985496260","heartbeat_front_addr":"172.21.15.194:6829/3985496260","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:21:17.278347+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:21:34.469282+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:21:51.139919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:22:09.126458+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:22:26.186801+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:22:43.178997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:23:01.109570+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:23:18.592191+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:23:35.685046+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:23:53.574820+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:24:10.987531+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:24:28.231206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.19:0/1047252857":"2024-01-25T14:20:02.911218+0000","172.21.15.19:6801/3758061162":"2024-01-25T14:20:02.911218+0000","172.21.15.19:6800/3758061162":"2024-01-25T14:20:02.911218+0000","172.21.15.19:6801/2754937713":"2024-01-25T14:19:39.853212+0000","172.21.15.19:0/4062420470":"2024-01-25T14:19:39.853212+0000","172.21.15.19:6800/2754937713":"2024-01-25T14:19:39.853212+0000","172.21.15.19:0/822613692":"2024-01-25T14:20:02.911218+0000","172.21.15.19:0/3970666946":"2024-01-25T14:19:39.853212+0000","172.21.15.19:0/1116053938":"2024-01-25T14:19:39.853212+0000","172.21.15.19:0/1456770388":"2024-01-25T14:20:02.911218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T14:25:08.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.778+0000 7f3d323db700 1 -- 172.21.15.19:0/680641486 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f3d140a0570 msgr2=0x7f3d140a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:08.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.778+0000 7f3d323db700 1 --2- 172.21.15.19:0/680641486 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f3d140a0570 0x7f3d140a2a30 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c009cb0 tx=0x7f3d1c009260 comp rx=0 tx=0).stop 2024-01-24T14:25:08.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.778+0000 7f3d323db700 1 -- 172.21.15.19:0/680641486 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3d2c104130 msgr2=0x7f3d2c197090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:08.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.778+0000 7f3d323db700 1 --2- 172.21.15.19:0/680641486 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3d2c104130 0x7f3d2c197090 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3d2000d980 tx=0x7f3d2000dc90 comp rx=0 tx=0).stop 2024-01-24T14:25:08.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.778+0000 7f3d323db700 1 -- 172.21.15.19:0/680641486 shutdown_connections 2024-01-24T14:25:08.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.778+0000 7f3d323db700 1 --2- 172.21.15.19:0/680641486 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3d2c106a60 0x7f3d2c19e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:08.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.778+0000 7f3d323db700 1 --2- 172.21.15.19:0/680641486 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3d2c101800 0x7f3d2c196b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:08.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.778+0000 7f3d323db700 1 --2- 172.21.15.19:0/680641486 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f3d140a0570 0x7f3d140a2a30 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:08.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.778+0000 7f3d323db700 1 --2- 172.21.15.19:0/680641486 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3d2c104130 0x7f3d2c197090 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:08.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.778+0000 7f3d323db700 1 -- 172.21.15.19:0/680641486 >> 172.21.15.19:0/680641486 conn(0x7f3d2c0fb170 msgr2=0x7f3d2c102490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:08.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.778+0000 7f3d323db700 1 -- 172.21.15.19:0/680641486 shutdown_connections 2024-01-24T14:25:08.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.778+0000 7f3d323db700 1 -- 172.21.15.19:0/680641486 wait complete. 2024-01-24T14:25:08.792 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T14:25:08.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:08 smithi123 ceph-mon[44966]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-01-24T14:25:08.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:08 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/2721565058' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-01-24T14:25:08.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:08 smithi123 ceph-mon[44966]: osdmap e81: 12 total, 12 up, 12 in 2024-01-24T14:25:08.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:08 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/3591851634' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-01-24T14:25:08.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:08 smithi123 ceph-mon[44966]: osdmap e82: 12 total, 12 up, 12 in 2024-01-24T14:25:08.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:08 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/3591851634' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-01-24T14:25:08.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:08 smithi123 ceph-mon[44966]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-01-24T14:25:08.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:08 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/2925370749' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T14:25:08.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:08 smithi194 ceph-mon[45450]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-01-24T14:25:08.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:08 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/2721565058' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-01-24T14:25:08.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:08 smithi194 ceph-mon[45450]: osdmap e81: 12 total, 12 up, 12 in 2024-01-24T14:25:08.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:08 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/3591851634' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-01-24T14:25:08.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:08 smithi194 ceph-mon[45450]: osdmap e82: 12 total, 12 up, 12 in 2024-01-24T14:25:08.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:08 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/3591851634' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-01-24T14:25:08.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:08 smithi194 ceph-mon[45450]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-01-24T14:25:08.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:08 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/2925370749' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T14:25:08.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.929+0000 7f0e4188c700 1 -- 172.21.15.19:0/2504538075 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0e3c1017c0 msgr2=0x7f0e3c103bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:08.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.929+0000 7f0e4188c700 1 --2- 172.21.15.19:0/2504538075 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0e3c1017c0 0x7f0e3c103bb0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c009990 tx=0x7f0e2c009ca0 comp rx=0 tx=0).stop 2024-01-24T14:25:08.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.930+0000 7f0e4188c700 1 -- 172.21.15.19:0/2504538075 shutdown_connections 2024-01-24T14:25:08.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.930+0000 7f0e4188c700 1 --2- 172.21.15.19:0/2504538075 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0e3c1017c0 0x7f0e3c103bb0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:08.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.930+0000 7f0e4188c700 1 --2- 172.21.15.19:0/2504538075 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e3c106a20 0x7f0e3c108e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:08.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.930+0000 7f0e4188c700 1 --2- 172.21.15.19:0/2504538075 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0e3c1040f0 0x7f0e3c1064e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:08.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.930+0000 7f0e4188c700 1 -- 172.21.15.19:0/2504538075 >> 172.21.15.19:0/2504538075 conn(0x7f0e3c0fb190 msgr2=0x7f0e3c0fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:08.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.930+0000 7f0e4188c700 1 -- 172.21.15.19:0/2504538075 shutdown_connections 2024-01-24T14:25:08.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.930+0000 7f0e4188c700 1 -- 172.21.15.19:0/2504538075 wait complete. 2024-01-24T14:25:08.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.930+0000 7f0e4188c700 1 Processor -- start 2024-01-24T14:25:08.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.930+0000 7f0e4188c700 1 -- start start 2024-01-24T14:25:08.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.931+0000 7f0e4188c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0e3c1017c0 0x7f0e3c196960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:08.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.931+0000 7f0e4188c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e3c1040f0 0x7f0e3c196ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:08.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.931+0000 7f0e4188c700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0e3c106a20 0x7f0e3c19df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:08.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.931+0000 7f0e4188c700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3c19e4e0 con 0x7f0e3c106a20 2024-01-24T14:25:08.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.931+0000 7f0e4188c700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3c19e620 con 0x7f0e3c1040f0 2024-01-24T14:25:08.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.931+0000 7f0e4188c700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3c19e7b0 con 0x7f0e3c1017c0 2024-01-24T14:25:08.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.931+0000 7f0e3b7fe700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0e3c106a20 0x7f0e3c19df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:08.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.931+0000 7f0e3b7fe700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0e3c106a20 0x7f0e3c19df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:32910/0 (socket says 172.21.15.19:32910) 2024-01-24T14:25:08.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.931+0000 7f0e3b7fe700 1 -- 172.21.15.19:0/3337297123 learned_addr learned my addr 172.21.15.19:0/3337297123 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:25:08.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.931+0000 7f0e3affd700 1 --2- 172.21.15.19:0/3337297123 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0e3c1017c0 0x7f0e3c196960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:08.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.931+0000 7f0e3a7fc700 1 --2- 172.21.15.19:0/3337297123 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e3c1040f0 0x7f0e3c196ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:08.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.931+0000 7f0e3b7fe700 1 -- 172.21.15.19:0/3337297123 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0e3c1017c0 msgr2=0x7f0e3c196960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:08.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.931+0000 7f0e3b7fe700 1 --2- 172.21.15.19:0/3337297123 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0e3c1017c0 0x7f0e3c196960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:08.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.931+0000 7f0e3b7fe700 1 -- 172.21.15.19:0/3337297123 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e3c1040f0 msgr2=0x7f0e3c196ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:08.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.931+0000 7f0e3b7fe700 1 --2- 172.21.15.19:0/3337297123 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e3c1040f0 0x7f0e3c196ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:08.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.931+0000 7f0e3b7fe700 1 -- 172.21.15.19:0/3337297123 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e2c0095f0 con 0x7f0e3c106a20 2024-01-24T14:25:08.935 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.931+0000 7f0e3b7fe700 1 --2- 172.21.15.19:0/3337297123 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0e3c106a20 0x7f0e3c19df10 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0e3400ef30 tx=0x7f0e3400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:08.935 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.931+0000 7f0e4088a700 1 -- 172.21.15.19:0/3337297123 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e34018070 con 0x7f0e3c106a20 2024-01-24T14:25:08.935 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.931+0000 7f0e4088a700 1 -- 172.21.15.19:0/3337297123 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0e3400f040 con 0x7f0e3c106a20 2024-01-24T14:25:08.935 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.931+0000 7f0e4188c700 1 -- 172.21.15.19:0/3337297123 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e3c19ea30 con 0x7f0e3c106a20 2024-01-24T14:25:08.935 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.932+0000 7f0e4088a700 1 -- 172.21.15.19:0/3337297123 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e3400cd00 con 0x7f0e3c106a20 2024-01-24T14:25:08.935 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.932+0000 7f0e4188c700 1 -- 172.21.15.19:0/3337297123 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e3c19eef0 con 0x7f0e3c106a20 2024-01-24T14:25:08.935 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.933+0000 7f0e4188c700 1 -- 172.21.15.19:0/3337297123 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e3c0fcd10 con 0x7f0e3c106a20 2024-01-24T14:25:08.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.934+0000 7f0e4088a700 1 -- 172.21.15.19:0/3337297123 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f0e34003a00 con 0x7f0e3c106a20 2024-01-24T14:25:08.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.934+0000 7f0e4088a700 1 --2- 172.21.15.19:0/3337297123 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f0e240a0570 0x7f0e240a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:08.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.934+0000 7f0e3affd700 1 --2- 172.21.15.19:0/3337297123 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f0e240a0570 0x7f0e240a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:08.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.934+0000 7f0e4088a700 1 -- 172.21.15.19:0/3337297123 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f0e340cb6d0 con 0x7f0e3c106a20 2024-01-24T14:25:08.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.935+0000 7f0e3affd700 1 --2- 172.21.15.19:0/3337297123 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f0e240a0570 0x7f0e240a2a30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c00b2e0 tx=0x7f0e2c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:08.939 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:08.939+0000 7f0e4088a700 1 -- 172.21.15.19:0/3337297123 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f0e34095110 con 0x7f0e3c106a20 2024-01-24T14:25:09.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.153+0000 7f0e4188c700 1 -- 172.21.15.19:0/3337297123 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e3c02cc70 con 0x7f0e3c106a20 2024-01-24T14:25:09.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.154+0000 7f0e4088a700 1 -- 172.21.15.19:0/3337297123 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+23997 (secure 0 0 0) 0x7f0e34098730 con 0x7f0e3c106a20 2024-01-24T14:25:09.155 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:25:09.155 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":82,"fsid":"654b0c8e-bac3-11ee-95b1-87774f69a715","created":"2024-01-24T14:19:10.699369+0000","modified":"2024-01-24T14:25:07.960902+0000","last_up_change":"2024-01-24T14:24:30.032711+0000","last_in_change":"2024-01-24T14:24:13.776095+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T14:21:53.453429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-24T14:25:01.967538+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-24T14:25:02.864396+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-01-24T14:25:05.950591+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"81380ec6-8e19-4eb4-8158-8fafaed8d7e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":2456786579},{"type":"v1","addr":"172.21.15.19:6803","nonce":2456786579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":2456786579},{"type":"v1","addr":"172.21.15.19:6805","nonce":2456786579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":2456786579},{"type":"v1","addr":"172.21.15.19:6809","nonce":2456786579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":2456786579},{"type":"v1","addr":"172.21.15.19:6807","nonce":2456786579}]},"public_addr":"172.21.15.19:6803/2456786579","cluster_addr":"172.21.15.19:6805/2456786579","heartbeat_back_addr":"172.21.15.19:6809/2456786579","heartbeat_front_addr":"172.21.15.19:6807/2456786579","state":["exists","up"]},{"osd":1,"uuid":"687f5036-a6d2-4aa1-86e7-7570e51ef179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":2407548889},{"type":"v1","addr":"172.21.15.123:6801","nonce":2407548889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":2407548889},{"type":"v1","addr":"172.21.15.123:6803","nonce":2407548889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":2407548889},{"type":"v1","addr":"172.21.15.123:6807","nonce":2407548889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":2407548889},{"type":"v1","addr":"172.21.15.123:6805","nonce":2407548889}]},"public_addr":"172.21.15.123:6801/2407548889","cluster_addr":"172.21.15.123:6803/2407548889","heartbeat_back_addr":"172.21.15.123:6807/2407548889","heartbeat_front_addr":"172.21.15.123:6805/2407548889","state":["exists","up"]},{"osd":2,"uuid":"3e810904-371d-4ca8-b43c-434957a2aa12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":3632698637},{"type":"v1","addr":"172.21.15.194:6801","nonce":3632698637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":3632698637},{"type":"v1","addr":"172.21.15.194:6803","nonce":3632698637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":3632698637},{"type":"v1","addr":"172.21.15.194:6807","nonce":3632698637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":3632698637},{"type":"v1","addr":"172.21.15.194:6805","nonce":3632698637}]},"public_addr":"172.21.15.194:6801/3632698637","cluster_addr":"172.21.15.194:6803/3632698637","heartbeat_back_addr":"172.21.15.194:6807/3632698637","heartbeat_front_addr":"172.21.15.194:6805/3632698637","state":["exists","up"]},{"osd":3,"uuid":"3665d6e0-010a-40bd-b0c6-682af56a0cbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":3868219159},{"type":"v1","addr":"172.21.15.19:6811","nonce":3868219159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3868219159},{"type":"v1","addr":"172.21.15.19:6813","nonce":3868219159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":3868219159},{"type":"v1","addr":"172.21.15.19:6817","nonce":3868219159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":3868219159},{"type":"v1","addr":"172.21.15.19:6815","nonce":3868219159}]},"public_addr":"172.21.15.19:6811/3868219159","cluster_addr":"172.21.15.19:6813/3868219159","heartbeat_back_addr":"172.21.15.19:6817/3868219159","heartbeat_front_addr":"172.21.15.19:6815/3868219159","state":["exists","up"]},{"osd":4,"uuid":"029fb28f-9af5-4239-8ee5-5ac5d04cd34b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":3611579809},{"type":"v1","addr":"172.21.15.123:6809","nonce":3611579809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":3611579809},{"type":"v1","addr":"172.21.15.123:6811","nonce":3611579809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":3611579809},{"type":"v1","addr":"172.21.15.123:6815","nonce":3611579809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":3611579809},{"type":"v1","addr":"172.21.15.123:6813","nonce":3611579809}]},"public_addr":"172.21.15.123:6809/3611579809","cluster_addr":"172.21.15.123:6811/3611579809","heartbeat_back_addr":"172.21.15.123:6815/3611579809","heartbeat_front_addr":"172.21.15.123:6813/3611579809","state":["exists","up"]},{"osd":5,"uuid":"6e97f23b-f803-4fed-a0a2-7aee7de4970f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":2483361871},{"type":"v1","addr":"172.21.15.194:6809","nonce":2483361871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":2483361871},{"type":"v1","addr":"172.21.15.194:6811","nonce":2483361871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":2483361871},{"type":"v1","addr":"172.21.15.194:6815","nonce":2483361871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":2483361871},{"type":"v1","addr":"172.21.15.194:6813","nonce":2483361871}]},"public_addr":"172.21.15.194:6809/2483361871","cluster_addr":"172.21.15.194:6811/2483361871","heartbeat_back_addr":"172.21.15.194:6815/2483361871","heartbeat_front_addr":"172.21.15.194:6813/2483361871","state":["exists","up"]},{"osd":6,"uuid":"c8c8ca2e-4cc6-4861-94b0-6211e4936c25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":1290513386},{"type":"v1","addr":"172.21.15.19:6819","nonce":1290513386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":1290513386},{"type":"v1","addr":"172.21.15.19:6821","nonce":1290513386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":1290513386},{"type":"v1","addr":"172.21.15.19:6825","nonce":1290513386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":1290513386},{"type":"v1","addr":"172.21.15.19:6823","nonce":1290513386}]},"public_addr":"172.21.15.19:6819/1290513386","cluster_addr":"172.21.15.19:6821/1290513386","heartbeat_back_addr":"172.21.15.19:6825/1290513386","heartbeat_front_addr":"172.21.15.19:6823/1290513386","state":["exists","up"]},{"osd":7,"uuid":"9aaea7a8-7bc3-4e32-9328-20a6c30a6d8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1391966446},{"type":"v1","addr":"172.21.15.123:6817","nonce":1391966446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1391966446},{"type":"v1","addr":"172.21.15.123:6819","nonce":1391966446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1391966446},{"type":"v1","addr":"172.21.15.123:6823","nonce":1391966446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1391966446},{"type":"v1","addr":"172.21.15.123:6821","nonce":1391966446}]},"public_addr":"172.21.15.123:6817/1391966446","cluster_addr":"172.21.15.123:6819/1391966446","heartbeat_back_addr":"172.21.15.123:6823/1391966446","heartbeat_front_addr":"172.21.15.123:6821/1391966446","state":["exists","up"]},{"osd":8,"uuid":"e918ec1b-7608-4f73-9067-29082ea63999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":2260577139},{"type":"v1","addr":"172.21.15.194:6817","nonce":2260577139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":2260577139},{"type":"v1","addr":"172.21.15.194:6819","nonce":2260577139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":2260577139},{"type":"v1","addr":"172.21.15.194:6823","nonce":2260577139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":2260577139},{"type":"v1","addr":"172.21.15.194:6821","nonce":2260577139}]},"public_addr":"172.21.15.194:6817/2260577139","cluster_addr":"172.21.15.194:6819/2260577139","heartbeat_back_addr":"172.21.15.194:6823/2260577139","heartbeat_front_addr":"172.21.15.194:6821/2260577139","state":["exists","up"]},{"osd":9,"uuid":"0fd76722-3319-4828-ab68-108607a4bd0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":4004843075},{"type":"v1","addr":"172.21.15.19:6827","nonce":4004843075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":4004843075},{"type":"v1","addr":"172.21.15.19:6829","nonce":4004843075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":4004843075},{"type":"v1","addr":"172.21.15.19:6833","nonce":4004843075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":4004843075},{"type":"v1","addr":"172.21.15.19:6831","nonce":4004843075}]},"public_addr":"172.21.15.19:6827/4004843075","cluster_addr":"172.21.15.19:6829/4004843075","heartbeat_back_addr":"172.21.15.19:6833/4004843075","heartbeat_front_addr":"172.21.15.19:6831/4004843075","state":["exists","up"]},{"osd":10,"uuid":"624bb8f5-6d72-43d5-9986-1b48b8a564ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2591845068},{"type":"v1","addr":"172.21.15.123:6825","nonce":2591845068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":2591845068},{"type":"v1","addr":"172.21.15.123:6827","nonce":2591845068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2591845068},{"type":"v1","addr":"172.21.15.123:6831","nonce":2591845068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2591845068},{"type":"v1","addr":"172.21.15.123:6829","nonce":2591845068}]},"public_addr":"172.21.15.123:6825/2591845068","cluster_addr":"172.21.15.123:6827/2591845068","heartbeat_back_addr":"172.21.15.123:6831/2591845068","heartbeat_front_addr":"172.21.15.123:6829/2591845068","state":["exists","up"]},{"osd":11,"uuid":"219a55b3-b01e-471d-90da-1a3043bf3e34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":3985496260},{"type":"v1","addr":"172.21.15.194:6825","nonce":3985496260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":3985496260},{"type":"v1","addr":"172.21.15.194:6827","nonce":3985496260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":3985496260},{"type":"v1","addr":"172.21.15.194:6831","nonce":3985496260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":3985496260},{"type":"v1","addr":"172.21.15.194:6829","nonce":3985496260}]},"public_addr":"172.21.15.194:6825/3985496260","cluster_addr":"172.21.15.194:6827/3985496260","heartbeat_back_addr":"172.21.15.194:6831/3985496260","heartbeat_front_addr":"172.21.15.194:6829/3985496260","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:21:17.278347+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:21:34.469282+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:21:51.139919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:22:09.126458+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:22:26.186801+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:22:43.178997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:23:01.109570+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:23:18.592191+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:23:35.685046+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:23:53.574820+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:24:10.987531+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:24:28.231206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.19:0/1047252857":"2024-01-25T14:20:02.911218+0000","172.21.15.19:6801/3758061162":"2024-01-25T14:20:02.911218+0000","172.21.15.19:6800/3758061162":"2024-01-25T14:20:02.911218+0000","172.21.15.19:6801/2754937713":"2024-01-25T14:19:39.853212+0000","172.21.15.19:0/4062420470":"2024-01-25T14:19:39.853212+0000","172.21.15.19:6800/2754937713":"2024-01-25T14:19:39.853212+0000","172.21.15.19:0/822613692":"2024-01-25T14:20:02.911218+0000","172.21.15.19:0/3970666946":"2024-01-25T14:19:39.853212+0000","172.21.15.19:0/1116053938":"2024-01-25T14:19:39.853212+0000","172.21.15.19:0/1456770388":"2024-01-25T14:20:02.911218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T14:25:09.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.157+0000 7f0e4188c700 1 -- 172.21.15.19:0/3337297123 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f0e240a0570 msgr2=0x7f0e240a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:09.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.157+0000 7f0e4188c700 1 --2- 172.21.15.19:0/3337297123 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f0e240a0570 0x7f0e240a2a30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c00b2e0 tx=0x7f0e2c005fb0 comp rx=0 tx=0).stop 2024-01-24T14:25:09.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.157+0000 7f0e4188c700 1 -- 172.21.15.19:0/3337297123 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0e3c106a20 msgr2=0x7f0e3c19df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:09.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.157+0000 7f0e4188c700 1 --2- 172.21.15.19:0/3337297123 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0e3c106a20 0x7f0e3c19df10 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0e3400ef30 tx=0x7f0e3400c3a0 comp rx=0 tx=0).stop 2024-01-24T14:25:09.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.157+0000 7f0e4188c700 1 -- 172.21.15.19:0/3337297123 shutdown_connections 2024-01-24T14:25:09.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.157+0000 7f0e4188c700 1 --2- 172.21.15.19:0/3337297123 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f0e240a0570 0x7f0e240a2a30 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:09.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.157+0000 7f0e4188c700 1 --2- 172.21.15.19:0/3337297123 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0e3c106a20 0x7f0e3c19df10 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:09.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.157+0000 7f0e4188c700 1 --2- 172.21.15.19:0/3337297123 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0e3c1017c0 0x7f0e3c196960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:09.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.157+0000 7f0e4188c700 1 --2- 172.21.15.19:0/3337297123 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0e3c1040f0 0x7f0e3c196ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:09.158 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.157+0000 7f0e4188c700 1 -- 172.21.15.19:0/3337297123 >> 172.21.15.19:0/3337297123 conn(0x7f0e3c0fb190 msgr2=0x7f0e3c102430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:09.158 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.157+0000 7f0e4188c700 1 -- 172.21.15.19:0/3337297123 shutdown_connections 2024-01-24T14:25:09.158 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.157+0000 7f0e4188c700 1 -- 172.21.15.19:0/3337297123 wait complete. 2024-01-24T14:25:09.185 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-24T14:25:09.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.324+0000 7f0dd0fb7700 1 -- 172.21.15.19:0/2621068605 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0dcc101fc0 msgr2=0x7f0dcc102420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:09.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.324+0000 7f0dd0fb7700 1 --2- 172.21.15.19:0/2621068605 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0dcc101fc0 0x7f0dcc102420 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0dc0009990 tx=0x7f0dc0009ca0 comp rx=0 tx=0).stop 2024-01-24T14:25:09.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.324+0000 7f0dd0fb7700 1 -- 172.21.15.19:0/2621068605 shutdown_connections 2024-01-24T14:25:09.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.324+0000 7f0dd0fb7700 1 --2- 172.21.15.19:0/2621068605 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0dcc102960 0x7f0dcc109210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:09.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.324+0000 7f0dd0fb7700 1 --2- 172.21.15.19:0/2621068605 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0dcc0feff0 0x7f0dcc0ff410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:09.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.324+0000 7f0dd0fb7700 1 --2- 172.21.15.19:0/2621068605 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0dcc101fc0 0x7f0dcc102420 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:09.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.324+0000 7f0dd0fb7700 1 -- 172.21.15.19:0/2621068605 >> 172.21.15.19:0/2621068605 conn(0x7f0dcc0fa900 msgr2=0x7f0dcc0fcd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:09.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.324+0000 7f0dd0fb7700 1 -- 172.21.15.19:0/2621068605 shutdown_connections 2024-01-24T14:25:09.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.324+0000 7f0dd0fb7700 1 -- 172.21.15.19:0/2621068605 wait complete. 2024-01-24T14:25:09.327 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.325+0000 7f0dd0fb7700 1 Processor -- start 2024-01-24T14:25:09.327 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.325+0000 7f0dd0fb7700 1 -- start start 2024-01-24T14:25:09.327 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.325+0000 7f0dd0fb7700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0dcc0feff0 0x7f0dcc101930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:09.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.325+0000 7f0dd0fb7700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0dcc101fc0 0x7f0dcc0fff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:09.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.325+0000 7f0dd0fb7700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0dcc102960 0x7f0dcc1004b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:09.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.325+0000 7f0dd0fb7700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dcc100c50 con 0x7f0dcc0feff0 2024-01-24T14:25:09.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.325+0000 7f0dd0fb7700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dcc100dc0 con 0x7f0dcc101fc0 2024-01-24T14:25:09.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.325+0000 7f0dd0fb7700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dcc1a09e0 con 0x7f0dcc102960 2024-01-24T14:25:09.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.325+0000 7f0dca59c700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0dcc0feff0 0x7f0dcc101930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:09.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.325+0000 7f0dca59c700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0dcc0feff0 0x7f0dcc101930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:32926/0 (socket says 172.21.15.19:32926) 2024-01-24T14:25:09.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.325+0000 7f0dca59c700 1 -- 172.21.15.19:0/3193467159 learned_addr learned my addr 172.21.15.19:0/3193467159 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:25:09.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.326+0000 7f0dc9d9b700 1 --2- 172.21.15.19:0/3193467159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0dcc101fc0 0x7f0dcc0fff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:09.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.326+0000 7f0dcad9d700 1 --2- 172.21.15.19:0/3193467159 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0dcc102960 0x7f0dcc1004b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:09.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.326+0000 7f0dca59c700 1 -- 172.21.15.19:0/3193467159 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0dcc102960 msgr2=0x7f0dcc1004b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:09.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.326+0000 7f0dca59c700 1 --2- 172.21.15.19:0/3193467159 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0dcc102960 0x7f0dcc1004b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:09.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.326+0000 7f0dca59c700 1 -- 172.21.15.19:0/3193467159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0dcc101fc0 msgr2=0x7f0dcc0fff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:09.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.326+0000 7f0dca59c700 1 --2- 172.21.15.19:0/3193467159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0dcc101fc0 0x7f0dcc0fff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:09.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.326+0000 7f0dca59c700 1 -- 172.21.15.19:0/3193467159 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc00095f0 con 0x7f0dcc0feff0 2024-01-24T14:25:09.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.326+0000 7f0dc9d9b700 1 --2- 172.21.15.19:0/3193467159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0dcc101fc0 0x7f0dcc0fff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T14:25:09.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.326+0000 7f0dcad9d700 1 --2- 172.21.15.19:0/3193467159 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0dcc102960 0x7f0dcc1004b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T14:25:09.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.326+0000 7f0dca59c700 1 --2- 172.21.15.19:0/3193467159 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0dcc0feff0 0x7f0dcc101930 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc00e970 tx=0x7f0dbc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:09.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.326+0000 7f0dbb7fe700 1 -- 172.21.15.19:0/3193467159 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc009790 con 0x7f0dcc0feff0 2024-01-24T14:25:09.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.326+0000 7f0dbb7fe700 1 -- 172.21.15.19:0/3193467159 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0dbc0044e0 con 0x7f0dcc0feff0 2024-01-24T14:25:09.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.326+0000 7f0dd0fb7700 1 -- 172.21.15.19:0/3193467159 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dcc1a0cc0 con 0x7f0dcc0feff0 2024-01-24T14:25:09.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.326+0000 7f0dbb7fe700 1 -- 172.21.15.19:0/3193467159 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc003e30 con 0x7f0dcc0feff0 2024-01-24T14:25:09.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.326+0000 7f0dd0fb7700 1 -- 172.21.15.19:0/3193467159 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dcc1a11d0 con 0x7f0dcc0feff0 2024-01-24T14:25:09.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.327+0000 7f0dd0fb7700 1 -- 172.21.15.19:0/3193467159 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dcc107d00 con 0x7f0dcc0feff0 2024-01-24T14:25:09.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.328+0000 7f0dbb7fe700 1 -- 172.21.15.19:0/3193467159 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f0dbc003f90 con 0x7f0dcc0feff0 2024-01-24T14:25:09.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.329+0000 7f0dbb7fe700 1 --2- 172.21.15.19:0/3193467159 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f0db40a05f0 0x7f0db40a2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:09.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.329+0000 7f0dc9d9b700 1 --2- 172.21.15.19:0/3193467159 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f0db40a05f0 0x7f0db40a2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:09.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.329+0000 7f0dbb7fe700 1 -- 172.21.15.19:0/3193467159 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f0dbc014070 con 0x7f0dcc0feff0 2024-01-24T14:25:09.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.330+0000 7f0dc9d9b700 1 --2- 172.21.15.19:0/3193467159 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f0db40a05f0 0x7f0db40a2ab0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0dc000b2e0 tx=0x7f0dc0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:09.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.334+0000 7f0dbb7fe700 1 -- 172.21.15.19:0/3193467159 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f0dbc0957f0 con 0x7f0dcc0feff0 2024-01-24T14:25:09.598 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.598+0000 7f0dd0fb7700 1 -- 172.21.15.19:0/3193467159 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0dcc02cc70 con 0x7f0dcc0feff0 2024-01-24T14:25:09.598 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.598+0000 7f0dbb7fe700 1 -- 172.21.15.19:0/3193467159 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+10813 (secure 0 0 0) 0x7f0dbc098e10 con 0x7f0dcc0feff0 2024-01-24T14:25:09.598 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:25:09.599 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14547,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6835/1060583512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6834","nonce":1060583512},{"type":"v1","addr":"172.21.15.19:6835","nonce":1060583512}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6837/2302964875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6836","nonce":2302964875},{"type":"v1","addr":"172.21.15.19:6837","nonce":2302964875}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14607,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6839/3376327575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6838","nonce":3376327575},{"type":"v1","addr":"172.21.15.19:6839","nonce":3376327575}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14619,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.19:6841/1702088084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6840","nonce":1702088084},{"type":"v1","addr":"172.21.15.19:6841","nonce":1702088084}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2647625404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2647625404},{"type":"v1","addr":"172.21.15.194:6833","nonce":2647625404}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24449,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/3396650565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":3396650565},{"type":"v1","addr":"172.21.15.123:6833","nonce":3396650565}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24485,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/2631676083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":2631676083},{"type":"v1","addr":"172.21.15.123:6835","nonce":2631676083}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24487,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6835/456696823","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":456696823},{"type":"v1","addr":"172.21.15.194:6835","nonce":456696823}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24499,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6837/2623846234","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":2623846234},{"type":"v1","addr":"172.21.15.194:6837","nonce":2623846234}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24506,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6837/491309336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6836","nonce":491309336},{"type":"v1","addr":"172.21.15.123:6837","nonce":491309336}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24514,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6839/810249223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":810249223},{"type":"v1","addr":"172.21.15.194:6839","nonce":810249223}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-24T14:25:03.484235+0000","modified":"2024-01-24T14:25:07.961163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24545},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.123:6839/4289194720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6838","nonce":4289194720},{"type":"v1","addr":"172.21.15.123:6839","nonce":4289194720}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-01-24T14:25:09.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.600+0000 7f0dd0fb7700 1 -- 172.21.15.19:0/3193467159 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f0db40a05f0 msgr2=0x7f0db40a2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:09.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.600+0000 7f0dd0fb7700 1 --2- 172.21.15.19:0/3193467159 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f0db40a05f0 0x7f0db40a2ab0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0dc000b2e0 tx=0x7f0dc0005fb0 comp rx=0 tx=0).stop 2024-01-24T14:25:09.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.600+0000 7f0dd0fb7700 1 -- 172.21.15.19:0/3193467159 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0dcc0feff0 msgr2=0x7f0dcc101930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:09.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.600+0000 7f0dd0fb7700 1 --2- 172.21.15.19:0/3193467159 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0dcc0feff0 0x7f0dcc101930 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc00e970 tx=0x7f0dbc00ed30 comp rx=0 tx=0).stop 2024-01-24T14:25:09.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.601+0000 7f0dd0fb7700 1 -- 172.21.15.19:0/3193467159 shutdown_connections 2024-01-24T14:25:09.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.601+0000 7f0dd0fb7700 1 --2- 172.21.15.19:0/3193467159 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f0db40a05f0 0x7f0db40a2ab0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:09.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.601+0000 7f0dd0fb7700 1 --2- 172.21.15.19:0/3193467159 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f0dcc0feff0 0x7f0dcc101930 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:09.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.601+0000 7f0dd0fb7700 1 --2- 172.21.15.19:0/3193467159 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0dcc102960 0x7f0dcc1004b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:09.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.601+0000 7f0dd0fb7700 1 --2- 172.21.15.19:0/3193467159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0dcc101fc0 0x7f0dcc0fff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:09.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.601+0000 7f0dd0fb7700 1 -- 172.21.15.19:0/3193467159 >> 172.21.15.19:0/3193467159 conn(0x7f0dcc0fa900 msgr2=0x7f0dcc0fcb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:09.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.601+0000 7f0dd0fb7700 1 -- 172.21.15.19:0/3193467159 shutdown_connections 2024-01-24T14:25:09.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.601+0000 7f0dd0fb7700 1 -- 172.21.15.19:0/3193467159 wait complete. 2024-01-24T14:25:09.602 INFO:teuthology.orchestra.run.smithi019.stderr:dumped fsmap epoch 16 2024-01-24T14:25:09.614 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T14:25:09.754 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.750+0000 7f7719b21700 1 -- 172.21.15.19:0/2553904769 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f77140ff590 msgr2=0x7f77141019c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:09.754 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.750+0000 7f7719b21700 1 --2- 172.21.15.19:0/2553904769 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f77140ff590 0x7f77141019c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7704009990 tx=0x7f7704009ca0 comp rx=0 tx=0).stop 2024-01-24T14:25:09.754 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.750+0000 7f7719b21700 1 -- 172.21.15.19:0/2553904769 shutdown_connections 2024-01-24T14:25:09.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.750+0000 7f7719b21700 1 --2- 172.21.15.19:0/2553904769 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f77140ff590 0x7f77141019c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:09.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.750+0000 7f7719b21700 1 --2- 172.21.15.19:0/2553904769 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77141048d0 0x7f7714104d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:09.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.750+0000 7f7719b21700 1 --2- 172.21.15.19:0/2553904769 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7714101f00 0x7f7714104390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:09.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.750+0000 7f7719b21700 1 -- 172.21.15.19:0/2553904769 >> 172.21.15.19:0/2553904769 conn(0x7f77140fb170 msgr2=0x7f77140fd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:09.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.750+0000 7f7719b21700 1 -- 172.21.15.19:0/2553904769 shutdown_connections 2024-01-24T14:25:09.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.750+0000 7f7719b21700 1 -- 172.21.15.19:0/2553904769 wait complete. 2024-01-24T14:25:09.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.751+0000 7f7719b21700 1 Processor -- start 2024-01-24T14:25:09.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.752+0000 7f7719b21700 1 -- start start 2024-01-24T14:25:09.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.752+0000 7f7719b21700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77140ff590 0x7f771419cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:09.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.752+0000 7f7719b21700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7714101f00 0x7f771419d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:09.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.752+0000 7f7719b21700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f77141048d0 0x7f77141a45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:09.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.752+0000 7f7719b21700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77141a4b70 con 0x7f7714101f00 2024-01-24T14:25:09.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.752+0000 7f7719b21700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77141a4cb0 con 0x7f77140ff590 2024-01-24T14:25:09.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.752+0000 7f7719b21700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77141a4e40 con 0x7f77141048d0 2024-01-24T14:25:09.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.752+0000 7f7712ffd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7714101f00 0x7f771419d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:09.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.752+0000 7f7712ffd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7714101f00 0x7f771419d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:32950/0 (socket says 172.21.15.19:32950) 2024-01-24T14:25:09.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.752+0000 7f7712ffd700 1 -- 172.21.15.19:0/142210274 learned_addr learned my addr 172.21.15.19:0/142210274 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:25:09.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.752+0000 7f77137fe700 1 --2- 172.21.15.19:0/142210274 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77140ff590 0x7f771419cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:09.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.752+0000 7f7713fff700 1 --2- 172.21.15.19:0/142210274 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f77141048d0 0x7f77141a45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:09.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.752+0000 7f7712ffd700 1 -- 172.21.15.19:0/142210274 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f77141048d0 msgr2=0x7f77141a45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:09.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.752+0000 7f7712ffd700 1 --2- 172.21.15.19:0/142210274 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f77141048d0 0x7f77141a45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:09.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.752+0000 7f7712ffd700 1 -- 172.21.15.19:0/142210274 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77140ff590 msgr2=0x7f771419cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:09.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.752+0000 7f7712ffd700 1 --2- 172.21.15.19:0/142210274 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77140ff590 0x7f771419cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:09.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.752+0000 7f7712ffd700 1 -- 172.21.15.19:0/142210274 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77040095f0 con 0x7f7714101f00 2024-01-24T14:25:09.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.753+0000 7f7712ffd700 1 --2- 172.21.15.19:0/142210274 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7714101f00 0x7f771419d530 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f770800e9a0 tx=0x7f770800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:09.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.753+0000 7f7710ff9700 1 -- 172.21.15.19:0/142210274 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7708009790 con 0x7f7714101f00 2024-01-24T14:25:09.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.753+0000 7f7719b21700 1 -- 172.21.15.19:0/142210274 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77141a5120 con 0x7f7714101f00 2024-01-24T14:25:09.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.753+0000 7f7710ff9700 1 -- 172.21.15.19:0/142210274 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7708010430 con 0x7f7714101f00 2024-01-24T14:25:09.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.753+0000 7f7710ff9700 1 -- 172.21.15.19:0/142210274 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7708018680 con 0x7f7714101f00 2024-01-24T14:25:09.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.753+0000 7f7719b21700 1 -- 172.21.15.19:0/142210274 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77141a5680 con 0x7f7714101f00 2024-01-24T14:25:09.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.754+0000 7f7719b21700 1 -- 172.21.15.19:0/142210274 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77140fccf0 con 0x7f7714101f00 2024-01-24T14:25:09.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.755+0000 7f7710ff9700 1 -- 172.21.15.19:0/142210274 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f77080105a0 con 0x7f7714101f00 2024-01-24T14:25:09.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.756+0000 7f7710ff9700 1 --2- 172.21.15.19:0/142210274 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f76fc0a0570 0x7f76fc0a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:09.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.756+0000 7f77137fe700 1 --2- 172.21.15.19:0/142210274 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f76fc0a0570 0x7f76fc0a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:09.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.756+0000 7f7710ff9700 1 -- 172.21.15.19:0/142210274 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f7708014070 con 0x7f7714101f00 2024-01-24T14:25:09.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.756+0000 7f77137fe700 1 --2- 172.21.15.19:0/142210274 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f76fc0a0570 0x7f76fc0a2a30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f770400b2e0 tx=0x7f7704005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:09.759 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.759+0000 7f7710ff9700 1 -- 172.21.15.19:0/142210274 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f77080958b0 con 0x7f7714101f00 2024-01-24T14:25:09.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:09 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/680641486' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T14:25:09.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:09 smithi123 ceph-mon[44966]: pgmap v225: 129 pgs: 4 creating+peering, 109 active+clean, 16 unknown; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 10 op/s 2024-01-24T14:25:09.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:09 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/3337297123' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T14:25:09.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:09 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/680641486' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T14:25:09.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:09 smithi194 ceph-mon[45450]: pgmap v225: 129 pgs: 4 creating+peering, 109 active+clean, 16 unknown; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 10 op/s 2024-01-24T14:25:09.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:09 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/3337297123' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T14:25:09.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.973+0000 7f7719b21700 1 -- 172.21.15.19:0/142210274 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f771402cc70 con 0x7f7714101f00 2024-01-24T14:25:09.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.975+0000 7f7710ff9700 1 -- 172.21.15.19:0/142210274 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+23997 (secure 0 0 0) 0x7f7708026090 con 0x7f7714101f00 2024-01-24T14:25:09.975 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:25:09.975 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":82,"fsid":"654b0c8e-bac3-11ee-95b1-87774f69a715","created":"2024-01-24T14:19:10.699369+0000","modified":"2024-01-24T14:25:07.960902+0000","last_up_change":"2024-01-24T14:24:30.032711+0000","last_in_change":"2024-01-24T14:24:13.776095+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T14:21:53.453429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-24T14:25:01.967538+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-24T14:25:02.864396+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-01-24T14:25:05.950591+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"81380ec6-8e19-4eb4-8158-8fafaed8d7e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":2456786579},{"type":"v1","addr":"172.21.15.19:6803","nonce":2456786579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":2456786579},{"type":"v1","addr":"172.21.15.19:6805","nonce":2456786579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":2456786579},{"type":"v1","addr":"172.21.15.19:6809","nonce":2456786579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":2456786579},{"type":"v1","addr":"172.21.15.19:6807","nonce":2456786579}]},"public_addr":"172.21.15.19:6803/2456786579","cluster_addr":"172.21.15.19:6805/2456786579","heartbeat_back_addr":"172.21.15.19:6809/2456786579","heartbeat_front_addr":"172.21.15.19:6807/2456786579","state":["exists","up"]},{"osd":1,"uuid":"687f5036-a6d2-4aa1-86e7-7570e51ef179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":2407548889},{"type":"v1","addr":"172.21.15.123:6801","nonce":2407548889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":2407548889},{"type":"v1","addr":"172.21.15.123:6803","nonce":2407548889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":2407548889},{"type":"v1","addr":"172.21.15.123:6807","nonce":2407548889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":2407548889},{"type":"v1","addr":"172.21.15.123:6805","nonce":2407548889}]},"public_addr":"172.21.15.123:6801/2407548889","cluster_addr":"172.21.15.123:6803/2407548889","heartbeat_back_addr":"172.21.15.123:6807/2407548889","heartbeat_front_addr":"172.21.15.123:6805/2407548889","state":["exists","up"]},{"osd":2,"uuid":"3e810904-371d-4ca8-b43c-434957a2aa12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":3632698637},{"type":"v1","addr":"172.21.15.194:6801","nonce":3632698637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":3632698637},{"type":"v1","addr":"172.21.15.194:6803","nonce":3632698637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":3632698637},{"type":"v1","addr":"172.21.15.194:6807","nonce":3632698637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":3632698637},{"type":"v1","addr":"172.21.15.194:6805","nonce":3632698637}]},"public_addr":"172.21.15.194:6801/3632698637","cluster_addr":"172.21.15.194:6803/3632698637","heartbeat_back_addr":"172.21.15.194:6807/3632698637","heartbeat_front_addr":"172.21.15.194:6805/3632698637","state":["exists","up"]},{"osd":3,"uuid":"3665d6e0-010a-40bd-b0c6-682af56a0cbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":3868219159},{"type":"v1","addr":"172.21.15.19:6811","nonce":3868219159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3868219159},{"type":"v1","addr":"172.21.15.19:6813","nonce":3868219159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":3868219159},{"type":"v1","addr":"172.21.15.19:6817","nonce":3868219159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":3868219159},{"type":"v1","addr":"172.21.15.19:6815","nonce":3868219159}]},"public_addr":"172.21.15.19:6811/3868219159","cluster_addr":"172.21.15.19:6813/3868219159","heartbeat_back_addr":"172.21.15.19:6817/3868219159","heartbeat_front_addr":"172.21.15.19:6815/3868219159","state":["exists","up"]},{"osd":4,"uuid":"029fb28f-9af5-4239-8ee5-5ac5d04cd34b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":3611579809},{"type":"v1","addr":"172.21.15.123:6809","nonce":3611579809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":3611579809},{"type":"v1","addr":"172.21.15.123:6811","nonce":3611579809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":3611579809},{"type":"v1","addr":"172.21.15.123:6815","nonce":3611579809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":3611579809},{"type":"v1","addr":"172.21.15.123:6813","nonce":3611579809}]},"public_addr":"172.21.15.123:6809/3611579809","cluster_addr":"172.21.15.123:6811/3611579809","heartbeat_back_addr":"172.21.15.123:6815/3611579809","heartbeat_front_addr":"172.21.15.123:6813/3611579809","state":["exists","up"]},{"osd":5,"uuid":"6e97f23b-f803-4fed-a0a2-7aee7de4970f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":2483361871},{"type":"v1","addr":"172.21.15.194:6809","nonce":2483361871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":2483361871},{"type":"v1","addr":"172.21.15.194:6811","nonce":2483361871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":2483361871},{"type":"v1","addr":"172.21.15.194:6815","nonce":2483361871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":2483361871},{"type":"v1","addr":"172.21.15.194:6813","nonce":2483361871}]},"public_addr":"172.21.15.194:6809/2483361871","cluster_addr":"172.21.15.194:6811/2483361871","heartbeat_back_addr":"172.21.15.194:6815/2483361871","heartbeat_front_addr":"172.21.15.194:6813/2483361871","state":["exists","up"]},{"osd":6,"uuid":"c8c8ca2e-4cc6-4861-94b0-6211e4936c25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":1290513386},{"type":"v1","addr":"172.21.15.19:6819","nonce":1290513386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":1290513386},{"type":"v1","addr":"172.21.15.19:6821","nonce":1290513386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":1290513386},{"type":"v1","addr":"172.21.15.19:6825","nonce":1290513386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":1290513386},{"type":"v1","addr":"172.21.15.19:6823","nonce":1290513386}]},"public_addr":"172.21.15.19:6819/1290513386","cluster_addr":"172.21.15.19:6821/1290513386","heartbeat_back_addr":"172.21.15.19:6825/1290513386","heartbeat_front_addr":"172.21.15.19:6823/1290513386","state":["exists","up"]},{"osd":7,"uuid":"9aaea7a8-7bc3-4e32-9328-20a6c30a6d8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1391966446},{"type":"v1","addr":"172.21.15.123:6817","nonce":1391966446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1391966446},{"type":"v1","addr":"172.21.15.123:6819","nonce":1391966446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1391966446},{"type":"v1","addr":"172.21.15.123:6823","nonce":1391966446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1391966446},{"type":"v1","addr":"172.21.15.123:6821","nonce":1391966446}]},"public_addr":"172.21.15.123:6817/1391966446","cluster_addr":"172.21.15.123:6819/1391966446","heartbeat_back_addr":"172.21.15.123:6823/1391966446","heartbeat_front_addr":"172.21.15.123:6821/1391966446","state":["exists","up"]},{"osd":8,"uuid":"e918ec1b-7608-4f73-9067-29082ea63999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":2260577139},{"type":"v1","addr":"172.21.15.194:6817","nonce":2260577139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":2260577139},{"type":"v1","addr":"172.21.15.194:6819","nonce":2260577139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":2260577139},{"type":"v1","addr":"172.21.15.194:6823","nonce":2260577139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":2260577139},{"type":"v1","addr":"172.21.15.194:6821","nonce":2260577139}]},"public_addr":"172.21.15.194:6817/2260577139","cluster_addr":"172.21.15.194:6819/2260577139","heartbeat_back_addr":"172.21.15.194:6823/2260577139","heartbeat_front_addr":"172.21.15.194:6821/2260577139","state":["exists","up"]},{"osd":9,"uuid":"0fd76722-3319-4828-ab68-108607a4bd0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":4004843075},{"type":"v1","addr":"172.21.15.19:6827","nonce":4004843075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":4004843075},{"type":"v1","addr":"172.21.15.19:6829","nonce":4004843075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":4004843075},{"type":"v1","addr":"172.21.15.19:6833","nonce":4004843075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":4004843075},{"type":"v1","addr":"172.21.15.19:6831","nonce":4004843075}]},"public_addr":"172.21.15.19:6827/4004843075","cluster_addr":"172.21.15.19:6829/4004843075","heartbeat_back_addr":"172.21.15.19:6833/4004843075","heartbeat_front_addr":"172.21.15.19:6831/4004843075","state":["exists","up"]},{"osd":10,"uuid":"624bb8f5-6d72-43d5-9986-1b48b8a564ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2591845068},{"type":"v1","addr":"172.21.15.123:6825","nonce":2591845068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":2591845068},{"type":"v1","addr":"172.21.15.123:6827","nonce":2591845068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2591845068},{"type":"v1","addr":"172.21.15.123:6831","nonce":2591845068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2591845068},{"type":"v1","addr":"172.21.15.123:6829","nonce":2591845068}]},"public_addr":"172.21.15.123:6825/2591845068","cluster_addr":"172.21.15.123:6827/2591845068","heartbeat_back_addr":"172.21.15.123:6831/2591845068","heartbeat_front_addr":"172.21.15.123:6829/2591845068","state":["exists","up"]},{"osd":11,"uuid":"219a55b3-b01e-471d-90da-1a3043bf3e34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":3985496260},{"type":"v1","addr":"172.21.15.194:6825","nonce":3985496260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":3985496260},{"type":"v1","addr":"172.21.15.194:6827","nonce":3985496260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":3985496260},{"type":"v1","addr":"172.21.15.194:6831","nonce":3985496260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":3985496260},{"type":"v1","addr":"172.21.15.194:6829","nonce":3985496260}]},"public_addr":"172.21.15.194:6825/3985496260","cluster_addr":"172.21.15.194:6827/3985496260","heartbeat_back_addr":"172.21.15.194:6831/3985496260","heartbeat_front_addr":"172.21.15.194:6829/3985496260","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:21:17.278347+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:21:34.469282+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:21:51.139919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:22:09.126458+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:22:26.186801+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:22:43.178997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:23:01.109570+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:23:18.592191+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:23:35.685046+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:23:53.574820+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:24:10.987531+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:24:28.231206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.19:0/1047252857":"2024-01-25T14:20:02.911218+0000","172.21.15.19:6801/3758061162":"2024-01-25T14:20:02.911218+0000","172.21.15.19:6800/3758061162":"2024-01-25T14:20:02.911218+0000","172.21.15.19:6801/2754937713":"2024-01-25T14:19:39.853212+0000","172.21.15.19:0/4062420470":"2024-01-25T14:19:39.853212+0000","172.21.15.19:6800/2754937713":"2024-01-25T14:19:39.853212+0000","172.21.15.19:0/822613692":"2024-01-25T14:20:02.911218+0000","172.21.15.19:0/3970666946":"2024-01-25T14:19:39.853212+0000","172.21.15.19:0/1116053938":"2024-01-25T14:19:39.853212+0000","172.21.15.19:0/1456770388":"2024-01-25T14:20:02.911218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T14:25:09.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.977+0000 7f7719b21700 1 -- 172.21.15.19:0/142210274 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f76fc0a0570 msgr2=0x7f76fc0a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:09.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.977+0000 7f7719b21700 1 --2- 172.21.15.19:0/142210274 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f76fc0a0570 0x7f76fc0a2a30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f770400b2e0 tx=0x7f7704005fb0 comp rx=0 tx=0).stop 2024-01-24T14:25:09.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.977+0000 7f7719b21700 1 -- 172.21.15.19:0/142210274 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7714101f00 msgr2=0x7f771419d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:09.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.977+0000 7f7719b21700 1 --2- 172.21.15.19:0/142210274 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7714101f00 0x7f771419d530 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f770800e9a0 tx=0x7f770800ecb0 comp rx=0 tx=0).stop 2024-01-24T14:25:09.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.977+0000 7f7719b21700 1 -- 172.21.15.19:0/142210274 shutdown_connections 2024-01-24T14:25:09.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.977+0000 7f7719b21700 1 --2- 172.21.15.19:0/142210274 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f77141048d0 0x7f77141a45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:09.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.977+0000 7f7719b21700 1 --2- 172.21.15.19:0/142210274 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77140ff590 0x7f771419cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:09.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.977+0000 7f7719b21700 1 --2- 172.21.15.19:0/142210274 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f76fc0a0570 0x7f76fc0a2a30 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:09.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.977+0000 7f7719b21700 1 --2- 172.21.15.19:0/142210274 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7714101f00 0x7f771419d530 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:09.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.977+0000 7f7719b21700 1 -- 172.21.15.19:0/142210274 >> 172.21.15.19:0/142210274 conn(0x7f77140fb170 msgr2=0x7f7714100200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:09.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.977+0000 7f7719b21700 1 -- 172.21.15.19:0/142210274 shutdown_connections 2024-01-24T14:25:09.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:09.977+0000 7f7719b21700 1 -- 172.21.15.19:0/142210274 wait complete. 2024-01-24T14:25:09.985 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:09 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/680641486' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T14:25:09.985 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:09 smithi019 ceph-mon[40663]: pgmap v225: 129 pgs: 4 creating+peering, 109 active+clean, 16 unknown; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 10 op/s 2024-01-24T14:25:09.985 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:09 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/3337297123' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T14:25:09.992 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-24T14:25:03.484235+0000', 'modified': '2024-01-24T14:25:07.961163+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24545}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24545': {'gid': 24545, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.123:6839/4289194720', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.123:6838', 'nonce': 4289194720}, {'type': 'v1', 'addr': '172.21.15.123:6839', 'nonce': 4289194720}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2024-01-24T14:25:09.992 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-24T14:25:09.992 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-01-24T14:25:10.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.130+0000 7f89414ad700 1 -- 172.21.15.19:0/2293791167 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f893c100d60 msgr2=0x7f893c101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:10.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.130+0000 7f89414ad700 1 --2- 172.21.15.19:0/2293791167 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f893c100d60 0x7f893c101180 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f892c0099c0 tx=0x7f892c009cd0 comp rx=0 tx=0).stop 2024-01-24T14:25:10.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.130+0000 7f89414ad700 1 -- 172.21.15.19:0/2293791167 shutdown_connections 2024-01-24T14:25:10.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.130+0000 7f89414ad700 1 --2- 172.21.15.19:0/2293791167 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f893c100d60 0x7f893c101180 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:10.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.130+0000 7f89414ad700 1 --2- 172.21.15.19:0/2293791167 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f893c1028f0 0x7f893c1091a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:10.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.130+0000 7f89414ad700 1 --2- 172.21.15.19:0/2293791167 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f893c101f50 0x7f893c1023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:10.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.130+0000 7f89414ad700 1 -- 172.21.15.19:0/2293791167 >> 172.21.15.19:0/2293791167 conn(0x7f893c0fc2d0 msgr2=0x7f893c0fe710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:10.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.130+0000 7f89414ad700 1 -- 172.21.15.19:0/2293791167 shutdown_connections 2024-01-24T14:25:10.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.130+0000 7f89414ad700 1 -- 172.21.15.19:0/2293791167 wait complete. 2024-01-24T14:25:10.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.130+0000 7f89414ad700 1 Processor -- start 2024-01-24T14:25:10.133 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.131+0000 7f89414ad700 1 -- start start 2024-01-24T14:25:10.133 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.131+0000 7f89414ad700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f893c100d60 0x7f893c196a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:10.133 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.131+0000 7f89414ad700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f893c101f50 0x7f893c196f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:10.133 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.131+0000 7f89414ad700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f893c1028f0 0x7f893c19dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:10.133 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.131+0000 7f89414ad700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893c19e5a0 con 0x7f893c100d60 2024-01-24T14:25:10.133 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.131+0000 7f89414ad700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893c19e6e0 con 0x7f893c101f50 2024-01-24T14:25:10.133 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.131+0000 7f89414ad700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893c19e870 con 0x7f893c1028f0 2024-01-24T14:25:10.133 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.131+0000 7f893affd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f893c100d60 0x7f893c196a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:10.133 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.131+0000 7f893affd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f893c100d60 0x7f893c196a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:32968/0 (socket says 172.21.15.19:32968) 2024-01-24T14:25:10.133 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.131+0000 7f893affd700 1 -- 172.21.15.19:0/3745885129 learned_addr learned my addr 172.21.15.19:0/3745885129 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:25:10.134 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.131+0000 7f893b7fe700 1 --2- 172.21.15.19:0/3745885129 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f893c1028f0 0x7f893c19dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:10.134 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.131+0000 7f893a7fc700 1 --2- 172.21.15.19:0/3745885129 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f893c101f50 0x7f893c196f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:10.134 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.131+0000 7f893affd700 1 -- 172.21.15.19:0/3745885129 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f893c1028f0 msgr2=0x7f893c19dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:10.134 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.131+0000 7f893affd700 1 --2- 172.21.15.19:0/3745885129 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f893c1028f0 0x7f893c19dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:10.134 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.131+0000 7f893affd700 1 -- 172.21.15.19:0/3745885129 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f893c101f50 msgr2=0x7f893c196f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:10.134 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.131+0000 7f893affd700 1 --2- 172.21.15.19:0/3745885129 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f893c101f50 0x7f893c196f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:10.134 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.131+0000 7f893affd700 1 -- 172.21.15.19:0/3745885129 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f892c0095f0 con 0x7f893c100d60 2024-01-24T14:25:10.134 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.131+0000 7f893affd700 1 --2- 172.21.15.19:0/3745885129 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f893c100d60 0x7f893c196a20 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f892c009990 tx=0x7f892c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:10.134 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.132+0000 7f8923fff700 1 -- 172.21.15.19:0/3745885129 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f892c01d070 con 0x7f893c100d60 2024-01-24T14:25:10.134 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.132+0000 7f8923fff700 1 -- 172.21.15.19:0/3745885129 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f892c004090 con 0x7f893c100d60 2024-01-24T14:25:10.135 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.132+0000 7f89414ad700 1 -- 172.21.15.19:0/3745885129 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f893c19eaf0 con 0x7f893c100d60 2024-01-24T14:25:10.135 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.132+0000 7f8923fff700 1 -- 172.21.15.19:0/3745885129 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f892c022d00 con 0x7f893c100d60 2024-01-24T14:25:10.135 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.132+0000 7f89414ad700 1 -- 172.21.15.19:0/3745885129 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f893c19eff0 con 0x7f893c100d60 2024-01-24T14:25:10.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.134+0000 7f8923fff700 1 -- 172.21.15.19:0/3745885129 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f892c022470 con 0x7f893c100d60 2024-01-24T14:25:10.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.135+0000 7f8923fff700 1 --2- 172.21.15.19:0/3745885129 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f89240a05a0 0x7f89240a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:10.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.135+0000 7f893a7fc700 1 --2- 172.21.15.19:0/3745885129 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f89240a05a0 0x7f89240a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:10.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.135+0000 7f8923fff700 1 -- 172.21.15.19:0/3745885129 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f892c0ccd40 con 0x7f893c100d60 2024-01-24T14:25:10.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.135+0000 7f893a7fc700 1 --2- 172.21.15.19:0/3745885129 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f89240a05a0 0x7f89240a2a60 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f893000bbd0 tx=0x7f893000b5b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:10.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.136+0000 7f894d641b80 1 Processor -- start 2024-01-24T14:25:10.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.136+0000 7f894d641b80 1 -- start start 2024-01-24T14:25:10.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.136+0000 7f894d641b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55863b5e1940 0x55863b5e1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:10.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.136+0000 7f894d641b80 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55863b5e22c0 0x55863b5e04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:10.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.136+0000 7f894d641b80 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55863b5e0ab0 0x55863b5e0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:10.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.136+0000 7f894d641b80 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x55863b54f470 con 0x55863b5e0ab0 2024-01-24T14:25:10.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.136+0000 7f894d641b80 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55863b533a50 con 0x55863b5e22c0 2024-01-24T14:25:10.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.136+0000 7f894d641b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x55863b531410 con 0x55863b5e1940 2024-01-24T14:25:10.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.136+0000 7f893b7fe700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55863b5e0ab0 0x55863b5e0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:10.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.136+0000 7f893b7fe700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55863b5e0ab0 0x55863b5e0f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:32976/0 (socket says 172.21.15.19:32976) 2024-01-24T14:25:10.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.136+0000 7f893b7fe700 1 -- 172.21.15.19:0/410481644 learned_addr learned my addr 172.21.15.19:0/410481644 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:25:10.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.136+0000 7f893affd700 1 --2- 172.21.15.19:0/410481644 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55863b5e1940 0x55863b5e1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:10.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.136+0000 7f893a7fc700 1 --2- 172.21.15.19:0/410481644 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55863b5e22c0 0x55863b5e04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:10.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.136+0000 7f893b7fe700 1 -- 172.21.15.19:0/410481644 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55863b5e1940 msgr2=0x55863b5e1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T14:25:10.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.136+0000 7f893b7fe700 1 --2- 172.21.15.19:0/410481644 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55863b5e1940 0x55863b5e1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:10.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.136+0000 7f893b7fe700 1 -- 172.21.15.19:0/410481644 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55863b5e22c0 msgr2=0x55863b5e04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T14:25:10.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.136+0000 7f893b7fe700 1 --2- 172.21.15.19:0/410481644 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55863b5e22c0 0x55863b5e04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:10.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.136+0000 7f893b7fe700 1 -- 172.21.15.19:0/410481644 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55863b3ae360 con 0x55863b5e0ab0 2024-01-24T14:25:10.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.137+0000 7f893b7fe700 1 --2- 172.21.15.19:0/410481644 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55863b5e0ab0 0x55863b5e0f30 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8934003fa0 tx=0x7f8934009ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecb293ae4d347f39 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:10.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.137+0000 7f89217fa700 1 -- 172.21.15.19:0/410481644 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8934019070 con 0x55863b5e0ab0 2024-01-24T14:25:10.139 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.137+0000 7f89217fa700 1 -- 172.21.15.19:0/410481644 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f893400f040 con 0x55863b5e0ab0 2024-01-24T14:25:10.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.137+0000 7f89217fa700 1 -- 172.21.15.19:0/410481644 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89340079a0 con 0x55863b5e0ab0 2024-01-24T14:25:10.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.138+0000 7f894d641b80 1 -- 172.21.15.19:0/410481644 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55863b5e0ab0 msgr2=0x55863b5e0f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:10.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.138+0000 7f894d641b80 1 --2- 172.21.15.19:0/410481644 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55863b5e0ab0 0x55863b5e0f30 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8934003fa0 tx=0x7f8934009ba0 comp rx=0 tx=0).stop 2024-01-24T14:25:10.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.138+0000 7f894d641b80 1 -- 172.21.15.19:0/410481644 shutdown_connections 2024-01-24T14:25:10.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.138+0000 7f894d641b80 1 --2- 172.21.15.19:0/410481644 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55863b5e0ab0 0x55863b5e0f30 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:10.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.138+0000 7f894d641b80 1 --2- 172.21.15.19:0/410481644 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55863b5e1940 0x55863b5e1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:10.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.138+0000 7f894d641b80 1 --2- 172.21.15.19:0/410481644 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55863b5e22c0 0x55863b5e04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:10.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.138+0000 7f894d641b80 1 -- 172.21.15.19:0/410481644 >> 172.21.15.19:0/410481644 conn(0x55863b53ad60 msgr2=0x55863b5d6f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:10.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.138+0000 7f894d641b80 1 -- 172.21.15.19:0/410481644 shutdown_connections 2024-01-24T14:25:10.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.138+0000 7f894d641b80 1 -- 172.21.15.19:0/410481644 wait complete. 2024-01-24T14:25:10.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.138+0000 7f894d641b80 1 -- start start 2024-01-24T14:25:10.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.138+0000 7f894d641b80 1 Processor -- start 2024-01-24T14:25:10.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.138+0000 7f894d641b80 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55863b5e0ab0 0x55863b5ea220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:10.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.138+0000 7f894d641b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55863b5e1940 0x55863b5ea760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:10.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.138+0000 7f894d641b80 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55863b5e22c0 0x55863b5f17a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:10.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.138+0000 7f894d641b80 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x55863b52cd00 con 0x55863b5e0ab0 2024-01-24T14:25:10.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.138+0000 7f894d641b80 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55863b4c8d30 con 0x55863b5e22c0 2024-01-24T14:25:10.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.138+0000 7f894d641b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x55863b4c89d0 con 0x55863b5e1940 2024-01-24T14:25:10.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.138+0000 7f894d641b80 10 client.0 _ll_register_callbacks cb 0x55863b4ca240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-24T14:25:10.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.138+0000 7f893affd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55863b5e0ab0 0x55863b5ea220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:10.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.139+0000 7f893affd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55863b5e0ab0 0x55863b5ea220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:32992/0 (socket says 172.21.15.19:32992) 2024-01-24T14:25:10.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.139+0000 7f893affd700 1 -- 172.21.15.19:0/2821178708 learned_addr learned my addr 172.21.15.19:0/2821178708 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:25:10.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.139+0000 7f893b7fe700 1 --2- 172.21.15.19:0/2821178708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55863b5e22c0 0x55863b5f17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:10.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.139+0000 7f893a7fc700 1 --2- 172.21.15.19:0/2821178708 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55863b5e1940 0x55863b5ea760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:10.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.139+0000 7f893affd700 1 -- 172.21.15.19:0/2821178708 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55863b5e1940 msgr2=0x55863b5ea760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T14:25:10.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.139+0000 7f893affd700 1 --2- 172.21.15.19:0/2821178708 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55863b5e1940 0x55863b5ea760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:10.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.139+0000 7f893affd700 1 -- 172.21.15.19:0/2821178708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55863b5e22c0 msgr2=0x55863b5f17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T14:25:10.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.139+0000 7f893affd700 1 --2- 172.21.15.19:0/2821178708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55863b5e22c0 0x55863b5f17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:10.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.139+0000 7f893affd700 1 -- 172.21.15.19:0/2821178708 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8934003bd0 con 0x55863b5e0ab0 2024-01-24T14:25:10.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.139+0000 7f893b7fe700 1 --2- 172.21.15.19:0/2821178708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55863b5e22c0 0x55863b5f17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T14:25:10.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.139+0000 7f893affd700 1 --2- 172.21.15.19:0/2821178708 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55863b5e0ab0 0x55863b5ea220 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f892c0ca9c0 tx=0x7f892c009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a0d51ccd8b5e8a2 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:10.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.139+0000 7f891eff5700 10 client.0 ms_handle_connect on v2:172.21.15.19:3300/0 2024-01-24T14:25:10.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.139+0000 7f891eff5700 1 -- 172.21.15.19:0/2821178708 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f892c0b1070 con 0x55863b5e0ab0 2024-01-24T14:25:10.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.139+0000 7f891eff5700 1 -- 172.21.15.19:0/2821178708 <== mon.0 v2:172.21.15.19:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f892c09e170 con 0x55863b5e0ab0 2024-01-24T14:25:10.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.139+0000 7f894d641b80 1 -- 172.21.15.19:0/2821178708 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55863b5cf140 con 0x55863b5e0ab0 2024-01-24T14:25:10.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.139+0000 7f891eff5700 1 -- 172.21.15.19:0/2821178708 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f892c0b4ea0 con 0x55863b5e0ab0 2024-01-24T14:25:10.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.139+0000 7f894d641b80 10 client.14739 fetch_fsmap learned FSMap version 16 2024-01-24T14:25:10.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.139+0000 7f894d641b80 1 -- 172.21.15.19:0/2821178708 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=4+}) v3 -- 0x55863b35ab10 con 0x55863b5e0ab0 2024-01-24T14:25:10.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.140+0000 7f891eff5700 1 -- 172.21.15.19:0/2821178708 <== mon.0 v2:172.21.15.19:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f892c064080 con 0x55863b5e0ab0 2024-01-24T14:25:10.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.140+0000 7f891eff5700 1 -- 172.21.15.19:0/2821178708 <== mon.0 v2:172.21.15.19:3300/0 5 ==== fsmap(e 16) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f892c0a20b0 con 0x55863b5e0ab0 2024-01-24T14:25:10.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.140+0000 7f894d641b80 10 client.14739 fetch_fsmap finished waiting for FSMap version 16 2024-01-24T14:25:10.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.140+0000 7f894d641b80 10 client.14739 resolve_mds: resolved name 'b' to daemon mds.b 2024-01-24T14:25:10.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.140+0000 7f894d641b80 20 client.14739 populate_metadata read hostname 'smithi019' 2024-01-24T14:25:10.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.140+0000 7f894d641b80 1 --2- 172.21.15.19:0/2821178708 >> [v2:172.21.15.123:6838/4289194720,v1:172.21.15.123:6839/4289194720] conn(0x55863b5f5100 0x55863b5f46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:10.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.140+0000 7f894d641b80 4 client.14739 mds_command: new command op to 24545 tid=0[{"prefix": "get_command_descriptions"}] 2024-01-24T14:25:10.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.140+0000 7f894d641b80 1 -- 172.21.15.19:0/2821178708 --> [v2:172.21.15.123:6838/4289194720,v1:172.21.15.123:6839/4289194720] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55863b525d80 con 0x55863b5f5100 2024-01-24T14:25:10.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.140+0000 7f893a7fc700 1 --2- 172.21.15.19:0/2821178708 >> [v2:172.21.15.123:6838/4289194720,v1:172.21.15.123:6839/4289194720] conn(0x55863b5f5100 0x55863b5f46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:10.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.140+0000 7f893a7fc700 1 --2- 172.21.15.19:0/2821178708 >> [v2:172.21.15.123:6838/4289194720,v1:172.21.15.123:6839/4289194720] conn(0x55863b5f5100 0x55863b5f46d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47aa7220efad30a6 server_cookie=7ee7808ba5b496e3 in_seq=0 out_seq=0 2024-01-24T14:25:10.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.140+0000 7f891eff5700 10 client.14739 ms_handle_connect on v2:172.21.15.123:6838/4289194720 2024-01-24T14:25:10.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.141+0000 7f891eff5700 1 -- 172.21.15.19:0/2821178708 <== mds.0 v2:172.21.15.123:6838/4289194720 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55863b525d80 con 0x55863b5f5100 2024-01-24T14:25:10.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.141+0000 7f891eff5700 10 client.14739 handle_command_reply: tid=0 2024-01-24T14:25:10.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.141+0000 7f894d641b80 1 client.14739 shutdown 2024-01-24T14:25:10.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.141+0000 7f894d641b80 1 -- 172.21.15.19:0/2821178708 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55863b5e0ab0 msgr2=0x55863b5ea220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:10.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.141+0000 7f894d641b80 1 --2- 172.21.15.19:0/2821178708 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55863b5e0ab0 0x55863b5ea220 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f892c0ca9c0 tx=0x7f892c009310 comp rx=0 tx=0).stop 2024-01-24T14:25:10.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.141+0000 7f894d641b80 1 -- 172.21.15.19:0/2821178708 shutdown_connections 2024-01-24T14:25:10.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.141+0000 7f894d641b80 1 --2- 172.21.15.19:0/2821178708 >> [v2:172.21.15.123:6838/4289194720,v1:172.21.15.123:6839/4289194720] conn(0x55863b5f5100 0x55863b5f46d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:10.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.141+0000 7f894d641b80 1 --2- 172.21.15.19:0/2821178708 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55863b5e0ab0 0x55863b5ea220 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:10.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.141+0000 7f894d641b80 1 --2- 172.21.15.19:0/2821178708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55863b5e22c0 0x55863b5f17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:10.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.141+0000 7f891eff5700 0 client.14739 ms_handle_reset on v2:172.21.15.123:6838/4289194720 2024-01-24T14:25:10.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.141+0000 7f894d641b80 1 --2- 172.21.15.19:0/2821178708 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55863b5e1940 0x55863b5ea760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:10.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.141+0000 7f894d641b80 1 -- 172.21.15.19:0/2821178708 >> 172.21.15.19:0/2821178708 conn(0x55863b53ad60 msgr2=0x55863b5d6f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:10.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.142+0000 7f894d641b80 1 -- 172.21.15.19:0/2821178708 shutdown_connections 2024-01-24T14:25:10.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.142+0000 7f894d641b80 1 -- 172.21.15.19:0/2821178708 wait complete. 2024-01-24T14:25:10.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.142+0000 7f894d641b80 20 client.14739 trim_cache size 0 max 16384 2024-01-24T14:25:10.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.160+0000 7f894d641b80 1 Processor -- start 2024-01-24T14:25:10.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.160+0000 7f894d641b80 1 -- start start 2024-01-24T14:25:10.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.160+0000 7f894d641b80 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55863b5e0ab0 0x55863b5d6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:10.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.160+0000 7f894d641b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55863b5e1940 0x55863b5fde70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:10.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.160+0000 7f894d641b80 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55863b5e22c0 0x55863b5e8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:10.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.160+0000 7f894d641b80 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x55863b3b0650 con 0x55863b5e22c0 2024-01-24T14:25:10.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.160+0000 7f894d641b80 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55863b4d6680 con 0x55863b5e0ab0 2024-01-24T14:25:10.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.160+0000 7f894d641b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x55863b531d80 con 0x55863b5e1940 2024-01-24T14:25:10.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.161+0000 7f893b7fe700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55863b5e22c0 0x55863b5e8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:10.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.161+0000 7f893affd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55863b5e0ab0 0x55863b5d6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:10.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.161+0000 7f893b7fe700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55863b5e22c0 0x55863b5e8ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:32998/0 (socket says 172.21.15.19:32998) 2024-01-24T14:25:10.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.161+0000 7f893b7fe700 1 -- 172.21.15.19:0/2673735100 learned_addr learned my addr 172.21.15.19:0/2673735100 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:25:10.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.161+0000 7f893a7fc700 1 --2- 172.21.15.19:0/2673735100 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55863b5e1940 0x55863b5fde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:10.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.161+0000 7f893b7fe700 1 -- 172.21.15.19:0/2673735100 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55863b5e1940 msgr2=0x55863b5fde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T14:25:10.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.161+0000 7f893b7fe700 1 --2- 172.21.15.19:0/2673735100 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55863b5e1940 0x55863b5fde70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:10.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.161+0000 7f893b7fe700 1 -- 172.21.15.19:0/2673735100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55863b5e0ab0 msgr2=0x55863b5d6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T14:25:10.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.161+0000 7f893b7fe700 1 --2- 172.21.15.19:0/2673735100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55863b5e0ab0 0x55863b5d6ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:10.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.161+0000 7f893b7fe700 1 -- 172.21.15.19:0/2673735100 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55863b4f2e50 con 0x55863b5e22c0 2024-01-24T14:25:10.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.161+0000 7f893b7fe700 1 --2- 172.21.15.19:0/2673735100 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55863b5e22c0 0x55863b5e8ff0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f89340093f0 tx=0x7f8934003ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=952fc7c2930bb0f server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:10.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.161+0000 7f8921ffb700 1 -- 172.21.15.19:0/2673735100 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8934019040 con 0x55863b5e22c0 2024-01-24T14:25:10.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.161+0000 7f8921ffb700 1 -- 172.21.15.19:0/2673735100 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8934005760 con 0x55863b5e22c0 2024-01-24T14:25:10.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.161+0000 7f8921ffb700 1 -- 172.21.15.19:0/2673735100 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89340156e0 con 0x55863b5e22c0 2024-01-24T14:25:10.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.162+0000 7f894d641b80 1 -- 172.21.15.19:0/2673735100 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55863b5e22c0 msgr2=0x55863b5e8ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:10.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.162+0000 7f894d641b80 1 --2- 172.21.15.19:0/2673735100 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55863b5e22c0 0x55863b5e8ff0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f89340093f0 tx=0x7f8934003ab0 comp rx=0 tx=0).stop 2024-01-24T14:25:10.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.162+0000 7f894d641b80 1 -- 172.21.15.19:0/2673735100 shutdown_connections 2024-01-24T14:25:10.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.162+0000 7f894d641b80 1 --2- 172.21.15.19:0/2673735100 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55863b5e22c0 0x55863b5e8ff0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:10.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.162+0000 7f894d641b80 1 --2- 172.21.15.19:0/2673735100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55863b5e0ab0 0x55863b5d6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:10.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.162+0000 7f894d641b80 1 --2- 172.21.15.19:0/2673735100 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55863b5e1940 0x55863b5fde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:10.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.162+0000 7f894d641b80 1 -- 172.21.15.19:0/2673735100 >> 172.21.15.19:0/2673735100 conn(0x55863b53ad60 msgr2=0x55863b5e03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:10.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.162+0000 7f894d641b80 1 -- 172.21.15.19:0/2673735100 shutdown_connections 2024-01-24T14:25:10.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.162+0000 7f894d641b80 1 -- 172.21.15.19:0/2673735100 wait complete. 2024-01-24T14:25:10.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.162+0000 7f894d641b80 1 -- start start 2024-01-24T14:25:10.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.162+0000 7f894d641b80 1 Processor -- start 2024-01-24T14:25:10.169 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.163+0000 7f894d641b80 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55863b5e0ab0 0x55863b5e6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:10.169 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.163+0000 7f894d641b80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55863b5e1940 0x55863b5fde70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:10.169 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.163+0000 7f894d641b80 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55863b5e22c0 0x55863b5e7050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:10.169 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.163+0000 7f894d641b80 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x55863b5e7790 con 0x55863b5e0ab0 2024-01-24T14:25:10.169 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.163+0000 7f894d641b80 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x55863b5e7900 con 0x55863b5e22c0 2024-01-24T14:25:10.169 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.163+0000 7f894d641b80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x55863b646c20 con 0x55863b5e1940 2024-01-24T14:25:10.169 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.163+0000 7f894d641b80 10 client.0 _ll_register_callbacks cb 0x55863b4ca240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-24T14:25:10.169 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.163+0000 7f893affd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55863b5e0ab0 0x55863b5e6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:10.169 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.163+0000 7f893affd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55863b5e0ab0 0x55863b5e6af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:33010/0 (socket says 172.21.15.19:33010) 2024-01-24T14:25:10.169 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.163+0000 7f893affd700 1 -- 172.21.15.19:0/3028879941 learned_addr learned my addr 172.21.15.19:0/3028879941 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:25:10.169 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.163+0000 7f893a7fc700 1 --2- 172.21.15.19:0/3028879941 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55863b5e1940 0x55863b5fde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:10.170 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.163+0000 7f893b7fe700 1 --2- 172.21.15.19:0/3028879941 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55863b5e22c0 0x55863b5e7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:10.170 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.163+0000 7f893affd700 1 -- 172.21.15.19:0/3028879941 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55863b5e1940 msgr2=0x55863b5fde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T14:25:10.170 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.163+0000 7f893affd700 1 --2- 172.21.15.19:0/3028879941 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55863b5e1940 0x55863b5fde70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:10.170 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.163+0000 7f893affd700 1 -- 172.21.15.19:0/3028879941 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55863b5e22c0 msgr2=0x55863b5e7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-24T14:25:10.170 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.163+0000 7f893affd700 1 --2- 172.21.15.19:0/3028879941 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55863b5e22c0 0x55863b5e7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:10.170 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.163+0000 7f893affd700 1 -- 172.21.15.19:0/3028879941 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55863b3ae360 con 0x55863b5e0ab0 2024-01-24T14:25:10.170 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.163+0000 7f893affd700 1 --2- 172.21.15.19:0/3028879941 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55863b5e0ab0 0x55863b5e6af0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f892c096fd0 tx=0x7f892c0042b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=444415241280d507 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:10.170 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.163+0000 7f891eff5700 10 client.0 ms_handle_connect on v2:172.21.15.19:3300/0 2024-01-24T14:25:10.170 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.163+0000 7f891eff5700 1 -- 172.21.15.19:0/3028879941 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f892c0b1040 con 0x55863b5e0ab0 2024-01-24T14:25:10.170 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.164+0000 7f891eff5700 1 -- 172.21.15.19:0/3028879941 <== mon.0 v2:172.21.15.19:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f892c09e170 con 0x55863b5e0ab0 2024-01-24T14:25:10.170 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.164+0000 7f894d641b80 1 -- 172.21.15.19:0/3028879941 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f893400f040 con 0x55863b5e0ab0 2024-01-24T14:25:10.170 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.164+0000 7f891eff5700 1 -- 172.21.15.19:0/3028879941 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f892c0968a0 con 0x55863b5e0ab0 2024-01-24T14:25:10.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.164+0000 7f894d641b80 10 client.14745 fetch_fsmap learned FSMap version 16 2024-01-24T14:25:10.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.164+0000 7f894d641b80 1 -- 172.21.15.19:0/3028879941 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=4+}) v3 -- 0x55863b3b2c80 con 0x55863b5e0ab0 2024-01-24T14:25:10.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.164+0000 7f891eff5700 1 -- 172.21.15.19:0/3028879941 <== mon.0 v2:172.21.15.19:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f892c07f080 con 0x55863b5e0ab0 2024-01-24T14:25:10.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.164+0000 7f891eff5700 1 -- 172.21.15.19:0/3028879941 <== mon.0 v2:172.21.15.19:3300/0 5 ==== fsmap(e 16) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f892c0caba0 con 0x55863b5e0ab0 2024-01-24T14:25:10.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.164+0000 7f894d641b80 10 client.14745 fetch_fsmap finished waiting for FSMap version 16 2024-01-24T14:25:10.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.164+0000 7f894d641b80 10 client.14745 resolve_mds: resolved name 'b' to daemon mds.b 2024-01-24T14:25:10.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.164+0000 7f894d641b80 20 client.14745 populate_metadata read hostname 'smithi019' 2024-01-24T14:25:10.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.164+0000 7f894d641b80 1 --2- 172.21.15.19:0/3028879941 >> [v2:172.21.15.123:6838/4289194720,v1:172.21.15.123:6839/4289194720] conn(0x55863b64b310 0x55863b647e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:10.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.164+0000 7f894d641b80 4 client.14745 mds_command: new command op to 24545 tid=0[{"prefix": "status"}] 2024-01-24T14:25:10.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.164+0000 7f894d641b80 1 -- 172.21.15.19:0/3028879941 --> [v2:172.21.15.123:6838/4289194720,v1:172.21.15.123:6839/4289194720] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55863b4ebf00 con 0x55863b64b310 2024-01-24T14:25:10.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.164+0000 7f893a7fc700 1 --2- 172.21.15.19:0/3028879941 >> [v2:172.21.15.123:6838/4289194720,v1:172.21.15.123:6839/4289194720] conn(0x55863b64b310 0x55863b647e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:10.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.165+0000 7f893a7fc700 1 --2- 172.21.15.19:0/3028879941 >> [v2:172.21.15.123:6838/4289194720,v1:172.21.15.123:6839/4289194720] conn(0x55863b64b310 0x55863b647e50 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7065d66c9a2bbfc server_cookie=925cf1895e798fec in_seq=0 out_seq=0 2024-01-24T14:25:10.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.165+0000 7f891eff5700 10 client.14745 ms_handle_connect on v2:172.21.15.123:6838/4289194720 2024-01-24T14:25:10.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.165+0000 7f891eff5700 1 -- 172.21.15.19:0/3028879941 <== mds.0 v2:172.21.15.123:6838/4289194720 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55863b4ebf00 con 0x55863b64b310 2024-01-24T14:25:10.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.165+0000 7f891eff5700 10 client.14745 handle_command_reply: tid=0 2024-01-24T14:25:10.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.165+0000 7f894d641b80 1 client.14745 shutdown 2024-01-24T14:25:10.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.165+0000 7f894d641b80 1 -- 172.21.15.19:0/3028879941 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55863b5e0ab0 msgr2=0x55863b5e6af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:10.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.165+0000 7f894d641b80 1 --2- 172.21.15.19:0/3028879941 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55863b5e0ab0 0x55863b5e6af0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f892c096fd0 tx=0x7f892c0042b0 comp rx=0 tx=0).stop 2024-01-24T14:25:10.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.165+0000 7f894d641b80 1 -- 172.21.15.19:0/3028879941 shutdown_connections 2024-01-24T14:25:10.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.165+0000 7f894d641b80 1 --2- 172.21.15.19:0/3028879941 >> [v2:172.21.15.123:6838/4289194720,v1:172.21.15.123:6839/4289194720] conn(0x55863b64b310 0x55863b647e50 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:10.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.165+0000 7f894d641b80 1 --2- 172.21.15.19:0/3028879941 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55863b5e0ab0 0x55863b5e6af0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:10.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.165+0000 7f891eff5700 0 client.14745 ms_handle_reset on v2:172.21.15.123:6838/4289194720 2024-01-24T14:25:10.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.165+0000 7f894d641b80 1 --2- 172.21.15.19:0/3028879941 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x55863b5e22c0 0x55863b5e7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:10.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.165+0000 7f894d641b80 1 --2- 172.21.15.19:0/3028879941 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55863b5e1940 0x55863b5fde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:10.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.165+0000 7f894d641b80 1 -- 172.21.15.19:0/3028879941 >> 172.21.15.19:0/3028879941 conn(0x55863b53ad60 msgr2=0x55863b5d6ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:10.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.165+0000 7f894d641b80 1 -- 172.21.15.19:0/3028879941 shutdown_connections 2024-01-24T14:25:10.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.165+0000 7f894d641b80 1 -- 172.21.15.19:0/3028879941 wait complete. 2024-01-24T14:25:10.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.165+0000 7f894d641b80 20 client.14745 trim_cache size 0 max 16384 2024-01-24T14:25:10.173 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-01-24T14:25:10.173 INFO:teuthology.orchestra.run.smithi019.stdout: "cluster_fsid": "654b0c8e-bac3-11ee-95b1-87774f69a715", 2024-01-24T14:25:10.174 INFO:teuthology.orchestra.run.smithi019.stdout: "whoami": 0, 2024-01-24T14:25:10.174 INFO:teuthology.orchestra.run.smithi019.stdout: "id": 24545, 2024-01-24T14:25:10.174 INFO:teuthology.orchestra.run.smithi019.stdout: "want_state": "up:active", 2024-01-24T14:25:10.174 INFO:teuthology.orchestra.run.smithi019.stdout: "state": "up:active", 2024-01-24T14:25:10.174 INFO:teuthology.orchestra.run.smithi019.stdout: "fs_name": "cephfs", 2024-01-24T14:25:10.174 INFO:teuthology.orchestra.run.smithi019.stdout: "rank_uptime": 6.6784271650000004, 2024-01-24T14:25:10.174 INFO:teuthology.orchestra.run.smithi019.stdout: "mdsmap_epoch": 16, 2024-01-24T14:25:10.174 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap_epoch": 82, 2024-01-24T14:25:10.174 INFO:teuthology.orchestra.run.smithi019.stdout: "osdmap_epoch_barrier": 81, 2024-01-24T14:25:10.174 INFO:teuthology.orchestra.run.smithi019.stdout: "uptime": 12.758110350000001 2024-01-24T14:25:10.174 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-01-24T14:25:10.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.166+0000 7f89414ad700 1 -- 172.21.15.19:0/3745885129 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f89240a05a0 msgr2=0x7f89240a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:10.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.166+0000 7f89414ad700 1 --2- 172.21.15.19:0/3745885129 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f89240a05a0 0x7f89240a2a60 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f893000bbd0 tx=0x7f893000b5b0 comp rx=0 tx=0).stop 2024-01-24T14:25:10.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.166+0000 7f89414ad700 1 -- 172.21.15.19:0/3745885129 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f893c100d60 msgr2=0x7f893c196a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:10.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.166+0000 7f89414ad700 1 --2- 172.21.15.19:0/3745885129 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f893c100d60 0x7f893c196a20 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f892c009990 tx=0x7f892c004730 comp rx=0 tx=0).stop 2024-01-24T14:25:10.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.166+0000 7f89414ad700 1 -- 172.21.15.19:0/3745885129 shutdown_connections 2024-01-24T14:25:10.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.166+0000 7f89414ad700 1 --2- 172.21.15.19:0/3745885129 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f89240a05a0 0x7f89240a2a60 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:10.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.166+0000 7f89414ad700 1 --2- 172.21.15.19:0/3745885129 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f893c100d60 0x7f893c196a20 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:10.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.166+0000 7f89414ad700 1 --2- 172.21.15.19:0/3745885129 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f893c1028f0 0x7f893c19dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:10.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.166+0000 7f89414ad700 1 --2- 172.21.15.19:0/3745885129 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f893c101f50 0x7f893c196f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:10.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.166+0000 7f89414ad700 1 -- 172.21.15.19:0/3745885129 >> 172.21.15.19:0/3745885129 conn(0x7f893c0fc2d0 msgr2=0x7f893c105440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:10.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.166+0000 7f89414ad700 1 -- 172.21.15.19:0/3745885129 shutdown_connections 2024-01-24T14:25:10.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:10.166+0000 7f89414ad700 1 -- 172.21.15.19:0/3745885129 wait complete. 2024-01-24T14:25:10.180 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-24T14:25:10.180 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-01-24T14:25:10.180 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:4e3182a65800aed6d4aa2f6b86da3cfd72188134 shell --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 -- ceph fs ls 2024-01-24T14:25:10.423 INFO:teuthology.orchestra.run.smithi019.stderr:Inferring config /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/mon.a/config 2024-01-24T14:25:10.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:10 smithi123 ceph-mon[44966]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-01-24T14:25:10.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:10 smithi123 ceph-mon[44966]: Cluster is now healthy 2024-01-24T14:25:10.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:10 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/3193467159' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T14:25:10.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:10 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/142210274' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T14:25:10.834 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:10 smithi019 ceph-mon[40663]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-01-24T14:25:10.834 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:10 smithi019 ceph-mon[40663]: Cluster is now healthy 2024-01-24T14:25:10.835 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:10 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/3193467159' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T14:25:10.835 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:10 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/142210274' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T14:25:10.843 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:10 smithi194 ceph-mon[45450]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-01-24T14:25:10.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:10 smithi194 ceph-mon[45450]: Cluster is now healthy 2024-01-24T14:25:10.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:10 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/3193467159' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-01-24T14:25:10.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:10 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/142210274' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T14:25:11.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.426+0000 7fd58f9c9700 1 -- 172.21.15.19:0/2625093580 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd588101000 msgr2=0x7fd5881013e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:11.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.426+0000 7fd58f9c9700 1 --2- 172.21.15.19:0/2625093580 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd588101000 0x7fd5881013e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd584009990 tx=0x7fd584009ca0 comp rx=0 tx=0).stop 2024-01-24T14:25:11.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.426+0000 7fd58f9c9700 1 -- 172.21.15.19:0/2625093580 shutdown_connections 2024-01-24T14:25:11.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.426+0000 7fd58f9c9700 1 --2- 172.21.15.19:0/2625093580 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd588101000 0x7fd5881013e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:11.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.426+0000 7fd58f9c9700 1 --2- 172.21.15.19:0/2625093580 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd5881060e0 0x7fd58810dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:11.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.426+0000 7fd58f9c9700 1 --2- 172.21.15.19:0/2625093580 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5881019b0 0x7fd588105a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:11.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.426+0000 7fd58f9c9700 1 -- 172.21.15.19:0/2625093580 >> 172.21.15.19:0/2625093580 conn(0x7fd5880fc870 msgr2=0x7fd5880fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:11.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.426+0000 7fd58f9c9700 1 -- 172.21.15.19:0/2625093580 shutdown_connections 2024-01-24T14:25:11.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.426+0000 7fd58f9c9700 1 -- 172.21.15.19:0/2625093580 wait complete. 2024-01-24T14:25:11.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.427+0000 7fd58f9c9700 1 Processor -- start 2024-01-24T14:25:11.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.427+0000 7fd58f9c9700 1 -- start start 2024-01-24T14:25:11.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.427+0000 7fd58f9c9700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd588101000 0x7fd588196d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:11.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.427+0000 7fd58f9c9700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd5881019b0 0x7fd588197270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:11.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.427+0000 7fd58f9c9700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5881060e0 0x7fd58819b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:11.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.427+0000 7fd58f9c9700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd58819bcf0 con 0x7fd5881019b0 2024-01-24T14:25:11.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.427+0000 7fd58f9c9700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd58819be60 con 0x7fd5881060e0 2024-01-24T14:25:11.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.427+0000 7fd58f9c9700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd58819c020 con 0x7fd588101000 2024-01-24T14:25:11.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.427+0000 7fd58cf64700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd5881019b0 0x7fd588197270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:11.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.428+0000 7fd58cf64700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd5881019b0 0x7fd588197270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:57648/0 (socket says 172.21.15.19:57648) 2024-01-24T14:25:11.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.428+0000 7fd58cf64700 1 -- 172.21.15.19:0/468855977 learned_addr learned my addr 172.21.15.19:0/468855977 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:25:11.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.428+0000 7fd58d765700 1 --2- 172.21.15.19:0/468855977 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd588101000 0x7fd588196d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:11.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.428+0000 7fd58df66700 1 --2- 172.21.15.19:0/468855977 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5881060e0 0x7fd58819b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:11.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.428+0000 7fd58cf64700 1 -- 172.21.15.19:0/468855977 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd588101000 msgr2=0x7fd588196d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:11.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.428+0000 7fd58cf64700 1 --2- 172.21.15.19:0/468855977 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd588101000 0x7fd588196d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:11.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.428+0000 7fd58cf64700 1 -- 172.21.15.19:0/468855977 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5881060e0 msgr2=0x7fd58819b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:11.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.428+0000 7fd58cf64700 1 --2- 172.21.15.19:0/468855977 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5881060e0 0x7fd58819b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:11.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.428+0000 7fd58cf64700 1 -- 172.21.15.19:0/468855977 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5840095f0 con 0x7fd5881019b0 2024-01-24T14:25:11.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.428+0000 7fd58df66700 1 --2- 172.21.15.19:0/468855977 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5881060e0 0x7fd58819b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T14:25:11.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.428+0000 7fd58cf64700 1 --2- 172.21.15.19:0/468855977 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd5881019b0 0x7fd588197270 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd57800c9e0 tx=0x7fd57800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:11.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.428+0000 7fd57e7fc700 1 -- 172.21.15.19:0/468855977 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd578007aa0 con 0x7fd5881019b0 2024-01-24T14:25:11.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.428+0000 7fd57e7fc700 1 -- 172.21.15.19:0/468855977 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd578007c00 con 0x7fd5881019b0 2024-01-24T14:25:11.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.428+0000 7fd58f9c9700 1 -- 172.21.15.19:0/468855977 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd58819c300 con 0x7fd5881019b0 2024-01-24T14:25:11.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.428+0000 7fd57e7fc700 1 -- 172.21.15.19:0/468855977 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd578018660 con 0x7fd5881019b0 2024-01-24T14:25:11.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.428+0000 7fd58f9c9700 1 -- 172.21.15.19:0/468855977 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5881a3b60 con 0x7fd5881019b0 2024-01-24T14:25:11.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.429+0000 7fd58f9c9700 1 -- 172.21.15.19:0/468855977 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd58804ea40 con 0x7fd5881019b0 2024-01-24T14:25:11.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.430+0000 7fd57e7fc700 1 -- 172.21.15.19:0/468855977 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7fd57800f450 con 0x7fd5881019b0 2024-01-24T14:25:11.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.431+0000 7fd57e7fc700 1 --2- 172.21.15.19:0/468855977 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fd5740a05a0 0x7fd5740a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:11.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.431+0000 7fd58d765700 1 --2- 172.21.15.19:0/468855977 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fd5740a05a0 0x7fd5740a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:11.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.431+0000 7fd57e7fc700 1 -- 172.21.15.19:0/468855977 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fd5780cba80 con 0x7fd5881019b0 2024-01-24T14:25:11.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.432+0000 7fd58d765700 1 --2- 172.21.15.19:0/468855977 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fd5740a05a0 0x7fd5740a2a60 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd58400b5c0 tx=0x7fd584005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:11.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.435+0000 7fd57e7fc700 1 -- 172.21.15.19:0/468855977 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fd578095430 con 0x7fd5881019b0 2024-01-24T14:25:11.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.694+0000 7fd58f9c9700 1 -- 172.21.15.19:0/468855977 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd588198090 con 0x7fd5881019b0 2024-01-24T14:25:11.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.695+0000 7fd57e7fc700 1 -- 172.21.15.19:0/468855977 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+88 (secure 0 0 0) 0x7fd578098a50 con 0x7fd5881019b0 2024-01-24T14:25:11.695 INFO:teuthology.orchestra.run.smithi019.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-01-24T14:25:11.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.697+0000 7fd58f9c9700 1 -- 172.21.15.19:0/468855977 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fd5740a05a0 msgr2=0x7fd5740a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:11.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.697+0000 7fd58f9c9700 1 --2- 172.21.15.19:0/468855977 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fd5740a05a0 0x7fd5740a2a60 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd58400b5c0 tx=0x7fd584005960 comp rx=0 tx=0).stop 2024-01-24T14:25:11.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.697+0000 7fd58f9c9700 1 -- 172.21.15.19:0/468855977 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd5881019b0 msgr2=0x7fd588197270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:11.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.697+0000 7fd58f9c9700 1 --2- 172.21.15.19:0/468855977 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd5881019b0 0x7fd588197270 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd57800c9e0 tx=0x7fd57800ccf0 comp rx=0 tx=0).stop 2024-01-24T14:25:11.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.697+0000 7fd58f9c9700 1 -- 172.21.15.19:0/468855977 shutdown_connections 2024-01-24T14:25:11.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.697+0000 7fd58f9c9700 1 --2- 172.21.15.19:0/468855977 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5881060e0 0x7fd58819b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:11.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.697+0000 7fd58f9c9700 1 --2- 172.21.15.19:0/468855977 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd588101000 0x7fd588196d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:11.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.697+0000 7fd58f9c9700 1 --2- 172.21.15.19:0/468855977 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fd5740a05a0 0x7fd5740a2a60 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:11.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.697+0000 7fd58f9c9700 1 --2- 172.21.15.19:0/468855977 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd5881019b0 0x7fd588197270 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:11.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.697+0000 7fd58f9c9700 1 -- 172.21.15.19:0/468855977 >> 172.21.15.19:0/468855977 conn(0x7fd5880fc870 msgr2=0x7fd5880fd2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:11.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.697+0000 7fd58f9c9700 1 -- 172.21.15.19:0/468855977 shutdown_connections 2024-01-24T14:25:11.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:11.697+0000 7fd58f9c9700 1 -- 172.21.15.19:0/468855977 wait complete. 2024-01-24T14:25:11.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:11 smithi123 ceph-mon[44966]: pgmap v226: 129 pgs: 4 creating+peering, 118 active+clean, 7 unknown; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 7 op/s 2024-01-24T14:25:11.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:11 smithi194 ceph-mon[45450]: pgmap v226: 129 pgs: 4 creating+peering, 118 active+clean, 7 unknown; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 7 op/s 2024-01-24T14:25:11.986 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:11 smithi019 ceph-mon[40663]: pgmap v226: 129 pgs: 4 creating+peering, 118 active+clean, 7 unknown; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 7 op/s 2024-01-24T14:25:12.203 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-01-24T14:25:12.203 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-24T14:25:12.203 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi019.front.sepia.ceph.com 2024-01-24T14:25:12.203 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-01-24T14:25:12.203 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-01-24T14:25:12.203 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-01-24T14:25:12.203 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-24T14:25:12.203 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-24T14:25:12.204 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-01-24T14:25:12.204 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T14:25:12.204 DEBUG:teuthology.orchestra.run.smithi019:> ip addr 2024-01-24T14:25:12.227 INFO:teuthology.orchestra.run.smithi019.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-24T14:25:12.227 INFO:teuthology.orchestra.run.smithi019.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-24T14:25:12.228 INFO:teuthology.orchestra.run.smithi019.stdout: inet 127.0.0.1/8 scope host lo 2024-01-24T14:25:12.228 INFO:teuthology.orchestra.run.smithi019.stdout: valid_lft forever preferred_lft forever 2024-01-24T14:25:12.228 INFO:teuthology.orchestra.run.smithi019.stdout: inet6 ::1/128 scope host 2024-01-24T14:25:12.228 INFO:teuthology.orchestra.run.smithi019.stdout: valid_lft forever preferred_lft forever 2024-01-24T14:25:12.228 INFO:teuthology.orchestra.run.smithi019.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T14:25:12.228 INFO:teuthology.orchestra.run.smithi019.stdout: link/ether 0c:c4:7a:6b:fd:12 brd ff:ff:ff:ff:ff:ff 2024-01-24T14:25:12.228 INFO:teuthology.orchestra.run.smithi019.stdout: altname enp7s0f0 2024-01-24T14:25:12.228 INFO:teuthology.orchestra.run.smithi019.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T14:25:12.228 INFO:teuthology.orchestra.run.smithi019.stdout: link/ether 0c:c4:7a:6b:fd:13 brd ff:ff:ff:ff:ff:ff 2024-01-24T14:25:12.228 INFO:teuthology.orchestra.run.smithi019.stdout: altname enp7s0f1 2024-01-24T14:25:12.228 INFO:teuthology.orchestra.run.smithi019.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-24T14:25:12.228 INFO:teuthology.orchestra.run.smithi019.stdout: link/ether 0c:c4:7a:bd:16:1e brd ff:ff:ff:ff:ff:ff 2024-01-24T14:25:12.228 INFO:teuthology.orchestra.run.smithi019.stdout: altname enp1s0f0 2024-01-24T14:25:12.229 INFO:teuthology.orchestra.run.smithi019.stdout: inet 172.21.15.19/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-24T14:25:12.229 INFO:teuthology.orchestra.run.smithi019.stdout: valid_lft 41939sec preferred_lft 41939sec 2024-01-24T14:25:12.229 INFO:teuthology.orchestra.run.smithi019.stdout: inet6 fe80::ec4:7aff:febd:161e/64 scope link 2024-01-24T14:25:12.229 INFO:teuthology.orchestra.run.smithi019.stdout: valid_lft forever preferred_lft forever 2024-01-24T14:25:12.229 INFO:teuthology.orchestra.run.smithi019.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-24T14:25:12.229 INFO:teuthology.orchestra.run.smithi019.stdout: link/ether 0c:c4:7a:bd:16:1f brd ff:ff:ff:ff:ff:ff 2024-01-24T14:25:12.229 INFO:teuthology.orchestra.run.smithi019.stdout: altname enp1s0f1 2024-01-24T14:25:12.229 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-01-24T14:25:12.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T14:25:12.230 DEBUG:teuthology.orchestra.run.smithi019:> (cd / && exec bash -c ' 2024-01-24T14:25:12.230 DEBUG:teuthology.orchestra.run.smithi019:> set -e 2024-01-24T14:25:12.230 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip link add name ceph-brx type bridge 2024-01-24T14:25:12.230 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip addr flush dev ceph-brx 2024-01-24T14:25:12.230 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip link set ceph-brx up 2024-01-24T14:25:12.230 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-01-24T14:25:12.230 DEBUG:teuthology.orchestra.run.smithi019:> ') 2024-01-24T14:25:12.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T14:25:12.415 DEBUG:teuthology.orchestra.run.smithi019:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-24T14:25:12.449 INFO:teuthology.orchestra.run.smithi019.stdout:1 2024-01-24T14:25:12.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T14:25:12.450 DEBUG:teuthology.orchestra.run.smithi019:> route 2024-01-24T14:25:12.525 INFO:teuthology.orchestra.run.smithi019.stdout:Kernel IP routing table 2024-01-24T14:25:12.526 INFO:teuthology.orchestra.run.smithi019.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-24T14:25:12.526 INFO:teuthology.orchestra.run.smithi019.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-24T14:25:12.526 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-24T14:25:12.526 INFO:teuthology.orchestra.run.smithi019.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-24T14:25:12.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T14:25:12.526 DEBUG:teuthology.orchestra.run.smithi019:> (cd / && exec bash -c ' 2024-01-24T14:25:12.527 DEBUG:teuthology.orchestra.run.smithi019:> set -e 2024-01-24T14:25:12.527 DEBUG:teuthology.orchestra.run.smithi019:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-24T14:25:12.527 DEBUG:teuthology.orchestra.run.smithi019:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-24T14:25:12.527 DEBUG:teuthology.orchestra.run.smithi019:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-24T14:25:12.527 DEBUG:teuthology.orchestra.run.smithi019:> ') 2024-01-24T14:25:12.619 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:12 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/468855977' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-01-24T14:25:12.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:12 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/468855977' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-01-24T14:25:12.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:12 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/468855977' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-01-24T14:25:12.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T14:25:12.939 DEBUG:teuthology.orchestra.run.smithi019:> ip netns list 2024-01-24T14:25:12.961 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T14:25:12.961 DEBUG:teuthology.orchestra.run.smithi019:> ip netns list-id 2024-01-24T14:25:13.023 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T14:25:13.023 DEBUG:teuthology.orchestra.run.smithi019:> (cd / && exec bash -c ' 2024-01-24T14:25:13.023 DEBUG:teuthology.orchestra.run.smithi019:> set -e 2024-01-24T14:25:13.023 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-01-24T14:25:13.023 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-01-24T14:25:13.023 DEBUG:teuthology.orchestra.run.smithi019:> ') 2024-01-24T14:25:13.137 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-01-24T14:25:13.137 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T14:25:13.137 DEBUG:teuthology.orchestra.run.smithi019:> (cd / && exec bash -c ' 2024-01-24T14:25:13.137 DEBUG:teuthology.orchestra.run.smithi019:> set -e 2024-01-24T14:25:13.137 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-01-24T14:25:13.137 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-01-24T14:25:13.137 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-01-24T14:25:13.137 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-01-24T14:25:13.137 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-01-24T14:25:13.137 DEBUG:teuthology.orchestra.run.smithi019:> ') 2024-01-24T14:25:13.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T14:25:13.361 DEBUG:teuthology.orchestra.run.smithi019:> (cd / && exec bash -c ' 2024-01-24T14:25:13.361 DEBUG:teuthology.orchestra.run.smithi019:> set -e 2024-01-24T14:25:13.361 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip link set brx.0 up 2024-01-24T14:25:13.362 DEBUG:teuthology.orchestra.run.smithi019:> sudo ip link set dev brx.0 master ceph-brx 2024-01-24T14:25:13.362 DEBUG:teuthology.orchestra.run.smithi019:> ') 2024-01-24T14:25:13.475 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-01-24T14:25:13.475 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T14:25:13.475 DEBUG:teuthology.orchestra.run.smithi019:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-01-24T14:25:13.495 INFO:teuthology.orchestra.run.smithi019.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-01-24T14:25:13.495 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-24T14:25:13.495 DEBUG:teuthology.orchestra.run.smithi019:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-01-24T14:25:13.555 DEBUG:teuthology.orchestra.run.smithi019:> sudo modprobe fuse 2024-01-24T14:25:13.630 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T14:25:13.630 DEBUG:teuthology.orchestra.run.smithi019:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T14:25:13.702 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:13 smithi019 ceph-mon[40663]: pgmap v227: 129 pgs: 129 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-01-24T14:25:13.706 INFO:teuthology.orchestra.run.smithi019.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T14:25:13.707 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T14:25:13.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T14:25:13.708 DEBUG:teuthology.orchestra.run.smithi019:> ls /sys/fs/fuse/connections 2024-01-24T14:25:13.770 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-24T14:25:13.770 DEBUG:teuthology.orchestra.run.smithi019:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-01-24T14:25:13.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:13 smithi123 ceph-mon[44966]: pgmap v227: 129 pgs: 129 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-01-24T14:25:13.813 DEBUG:teuthology.orchestra.run.smithi019:> sudo modprobe fuse 2024-01-24T14:25:13.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:13 smithi194 ceph-mon[45450]: pgmap v227: 129 pgs: 129 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-01-24T14:25:13.849 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-01-24T14:25:13.849 DEBUG:teuthology.orchestra.run.smithi019:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-01-24T14:25:13.920 INFO:teuthology.orchestra.run.smithi019.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-01-24T14:25:13.921 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-01-24T14:25:13.922 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T14:25:13.922 DEBUG:teuthology.orchestra.run.smithi019:> ls /sys/fs/fuse/connections 2024-01-24T14:25:13.923 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi019.stderr:2024-01-24T14:25:13.923+0000 7f2d8d8235c0 -1 init, newargv = 0x55dd58205b80 newargc=15 2024-01-24T14:25:13.923 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi019.stderr:ceph-fuse[95906]: starting ceph client 2024-01-24T14:25:13.930 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi019.stderr:ceph-fuse[95906]: starting fuse 2024-01-24T14:25:13.943 INFO:teuthology.orchestra.run.smithi019.stdout:124 2024-01-24T14:25:13.943 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [124] 2024-01-24T14:25:13.944 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-01-24T14:25:13.944 DEBUG:teuthology.orchestra.run.smithi019:> import glob 2024-01-24T14:25:13.944 DEBUG:teuthology.orchestra.run.smithi019:> import re 2024-01-24T14:25:13.944 DEBUG:teuthology.orchestra.run.smithi019:> import os 2024-01-24T14:25:13.944 DEBUG:teuthology.orchestra.run.smithi019:> import subprocess 2024-01-24T14:25:13.944 DEBUG:teuthology.orchestra.run.smithi019:> 2024-01-24T14:25:13.944 DEBUG:teuthology.orchestra.run.smithi019:> def _find_admin_socket(client_name): 2024-01-24T14:25:13.944 DEBUG:teuthology.orchestra.run.smithi019:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-01-24T14:25:13.944 DEBUG:teuthology.orchestra.run.smithi019:> files = glob.glob(asok_path) 2024-01-24T14:25:13.944 DEBUG:teuthology.orchestra.run.smithi019:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-01-24T14:25:13.944 DEBUG:teuthology.orchestra.run.smithi019:> 2024-01-24T14:25:13.944 DEBUG:teuthology.orchestra.run.smithi019:> # Given a non-glob path, it better be there 2024-01-24T14:25:13.944 DEBUG:teuthology.orchestra.run.smithi019:> if "*" not in asok_path: 2024-01-24T14:25:13.944 DEBUG:teuthology.orchestra.run.smithi019:> assert(len(files) == 1) 2024-01-24T14:25:13.945 DEBUG:teuthology.orchestra.run.smithi019:> return files[0] 2024-01-24T14:25:13.945 DEBUG:teuthology.orchestra.run.smithi019:> 2024-01-24T14:25:13.945 DEBUG:teuthology.orchestra.run.smithi019:> for f in files: 2024-01-24T14:25:13.945 DEBUG:teuthology.orchestra.run.smithi019:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-24T14:25:13.945 DEBUG:teuthology.orchestra.run.smithi019:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-24T14:25:13.945 DEBUG:teuthology.orchestra.run.smithi019:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-24T14:25:13.945 DEBUG:teuthology.orchestra.run.smithi019:> contents = proc_f.read() 2024-01-24T14:25:13.945 DEBUG:teuthology.orchestra.run.smithi019:> if mountpoint in contents: 2024-01-24T14:25:13.945 DEBUG:teuthology.orchestra.run.smithi019:> return f 2024-01-24T14:25:13.945 DEBUG:teuthology.orchestra.run.smithi019:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-24T14:25:13.945 DEBUG:teuthology.orchestra.run.smithi019:> 2024-01-24T14:25:13.945 DEBUG:teuthology.orchestra.run.smithi019:> print(_find_admin_socket("client.admin")) 2024-01-24T14:25:13.945 DEBUG:teuthology.orchestra.run.smithi019:> ' 2024-01-24T14:25:14.081 INFO:teuthology.orchestra.run.smithi019.stdout:/var/run/ceph/ceph-client.admin.95906.asok 2024-01-24T14:25:14.263 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.95906.asok 2024-01-24T14:25:14.263 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T14:25:14.264 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.95906.asok status 2024-01-24T14:25:14.443 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-01-24T14:25:14.444 INFO:teuthology.orchestra.run.smithi019.stdout: "metadata": { 2024-01-24T14:25:14.444 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph_sha1": "4e3182a65800aed6d4aa2f6b86da3cfd72188134", 2024-01-24T14:25:14.444 INFO:teuthology.orchestra.run.smithi019.stdout: "ceph_version": "ceph version 18.2.1-341-g4e3182a6 (4e3182a65800aed6d4aa2f6b86da3cfd72188134) reef (stable)", 2024-01-24T14:25:14.444 INFO:teuthology.orchestra.run.smithi019.stdout: "entity_id": "admin", 2024-01-24T14:25:14.444 INFO:teuthology.orchestra.run.smithi019.stdout: "hostname": "smithi019", 2024-01-24T14:25:14.444 INFO:teuthology.orchestra.run.smithi019.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-01-24T14:25:14.444 INFO:teuthology.orchestra.run.smithi019.stdout: "pid": "95906", 2024-01-24T14:25:14.444 INFO:teuthology.orchestra.run.smithi019.stdout: "root": "/" 2024-01-24T14:25:14.444 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-01-24T14:25:14.444 INFO:teuthology.orchestra.run.smithi019.stdout: "dentry_count": 0, 2024-01-24T14:25:14.444 INFO:teuthology.orchestra.run.smithi019.stdout: "dentry_pinned_count": 0, 2024-01-24T14:25:14.444 INFO:teuthology.orchestra.run.smithi019.stdout: "id": 14757, 2024-01-24T14:25:14.445 INFO:teuthology.orchestra.run.smithi019.stdout: "inst": { 2024-01-24T14:25:14.445 INFO:teuthology.orchestra.run.smithi019.stdout: "name": { 2024-01-24T14:25:14.445 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "client", 2024-01-24T14:25:14.445 INFO:teuthology.orchestra.run.smithi019.stdout: "num": 14757 2024-01-24T14:25:14.445 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-01-24T14:25:14.445 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": { 2024-01-24T14:25:14.445 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-01-24T14:25:14.445 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "192.168.0.1:0", 2024-01-24T14:25:14.445 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 98390207 2024-01-24T14:25:14.445 INFO:teuthology.orchestra.run.smithi019.stdout: } 2024-01-24T14:25:14.445 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-01-24T14:25:14.445 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": { 2024-01-24T14:25:14.445 INFO:teuthology.orchestra.run.smithi019.stdout: "type": "v1", 2024-01-24T14:25:14.445 INFO:teuthology.orchestra.run.smithi019.stdout: "addr": "192.168.0.1:0", 2024-01-24T14:25:14.445 INFO:teuthology.orchestra.run.smithi019.stdout: "nonce": 98390207 2024-01-24T14:25:14.446 INFO:teuthology.orchestra.run.smithi019.stdout: }, 2024-01-24T14:25:14.446 INFO:teuthology.orchestra.run.smithi019.stdout: "inst_str": "client.14757 192.168.0.1:0/98390207", 2024-01-24T14:25:14.446 INFO:teuthology.orchestra.run.smithi019.stdout: "addr_str": "192.168.0.1:0/98390207", 2024-01-24T14:25:14.446 INFO:teuthology.orchestra.run.smithi019.stdout: "inode_count": 1, 2024-01-24T14:25:14.446 INFO:teuthology.orchestra.run.smithi019.stdout: "mds_epoch": 16, 2024-01-24T14:25:14.446 INFO:teuthology.orchestra.run.smithi019.stdout: "osd_epoch": 82, 2024-01-24T14:25:14.446 INFO:teuthology.orchestra.run.smithi019.stdout: "osd_epoch_barrier": 0, 2024-01-24T14:25:14.446 INFO:teuthology.orchestra.run.smithi019.stdout: "blocklisted": false, 2024-01-24T14:25:14.446 INFO:teuthology.orchestra.run.smithi019.stdout: "fs_name": "cephfs" 2024-01-24T14:25:14.446 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-01-24T14:25:14.455 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T14:25:14.455 DEBUG:teuthology.orchestra.run.smithi019:> stat --file-system '--printf=%T 2024-01-24T14:25:14.456 DEBUG:teuthology.orchestra.run.smithi019:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-01-24T14:25:14.475 INFO:teuthology.orchestra.run.smithi019.stdout:fuseblk 2024-01-24T14:25:14.475 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-01-24T14:25:14.475 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T14:25:14.475 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-01-24T14:25:14.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T14:25:14.550 DEBUG:teuthology.orchestra.run.smithi019:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-01-24T14:25:14.675 INFO:teuthology.orchestra.run.smithi019.stdout:# file: . 2024-01-24T14:25:14.675 INFO:teuthology.orchestra.run.smithi019.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-01-24T14:25:14.675 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:25:14.676 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-01-24T14:25:14.676 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T14:25:14.699 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T14:25:14.887 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:14.885+0000 7fe0d32e6700 1 -- 172.21.15.19:0/1047184314 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe0cc101fc0 msgr2=0x7fe0cc102420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:14.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:14.885+0000 7fe0d32e6700 1 --2- 172.21.15.19:0/1047184314 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe0cc101fc0 0x7fe0cc102420 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe0c8009990 tx=0x7fe0c8009ca0 comp rx=0 tx=0).stop 2024-01-24T14:25:14.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:14.885+0000 7fe0d32e6700 1 -- 172.21.15.19:0/1047184314 shutdown_connections 2024-01-24T14:25:14.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:14.885+0000 7fe0d32e6700 1 --2- 172.21.15.19:0/1047184314 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe0cc102960 0x7fe0cc109210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:14.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:14.885+0000 7fe0d32e6700 1 --2- 172.21.15.19:0/1047184314 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0cc0feff0 0x7fe0cc0ff410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:14.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:14.885+0000 7fe0d32e6700 1 --2- 172.21.15.19:0/1047184314 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe0cc101fc0 0x7fe0cc102420 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:14.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:14.885+0000 7fe0d32e6700 1 -- 172.21.15.19:0/1047184314 >> 172.21.15.19:0/1047184314 conn(0x7fe0cc0fa900 msgr2=0x7fe0cc0fcd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:14.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:14.886+0000 7fe0d32e6700 1 -- 172.21.15.19:0/1047184314 shutdown_connections 2024-01-24T14:25:14.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:14.886+0000 7fe0d32e6700 1 -- 172.21.15.19:0/1047184314 wait complete. 2024-01-24T14:25:14.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:14.886+0000 7fe0d32e6700 1 Processor -- start 2024-01-24T14:25:14.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:14.886+0000 7fe0d32e6700 1 -- start start 2024-01-24T14:25:14.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:14.886+0000 7fe0d32e6700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe0cc0feff0 0x7fe0cc1969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:14.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:14.886+0000 7fe0d32e6700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe0cc101fc0 0x7fe0cc196ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:14.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:14.886+0000 7fe0d32e6700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0cc102960 0x7fe0cc19df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:14.889 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:14.886+0000 7fe0d32e6700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0cc19e520 con 0x7fe0cc101fc0 2024-01-24T14:25:14.889 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:14.886+0000 7fe0d32e6700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0cc19e660 con 0x7fe0cc102960 2024-01-24T14:25:14.889 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:14.886+0000 7fe0d32e6700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0cc19e7f0 con 0x7fe0cc0feff0 2024-01-24T14:25:14.889 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:14.887+0000 7fe0d0881700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe0cc101fc0 0x7fe0cc196ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:14.889 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:14.887+0000 7fe0d1082700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe0cc0feff0 0x7fe0cc1969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:14.889 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:14.887+0000 7fe0d0881700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe0cc101fc0 0x7fe0cc196ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:57670/0 (socket says 172.21.15.19:57670) 2024-01-24T14:25:14.889 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:14.887+0000 7fe0d0881700 1 -- 172.21.15.19:0/3529335714 learned_addr learned my addr 172.21.15.19:0/3529335714 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:25:14.889 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:14.887+0000 7fe0d1883700 1 --2- 172.21.15.19:0/3529335714 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0cc102960 0x7fe0cc19df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:14.889 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:14.887+0000 7fe0d0881700 1 -- 172.21.15.19:0/3529335714 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe0cc0feff0 msgr2=0x7fe0cc1969a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:14.889 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:14.887+0000 7fe0d0881700 1 --2- 172.21.15.19:0/3529335714 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe0cc0feff0 0x7fe0cc1969a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:14.889 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:14.887+0000 7fe0d0881700 1 -- 172.21.15.19:0/3529335714 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0cc102960 msgr2=0x7fe0cc19df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:14.889 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:14.887+0000 7fe0d0881700 1 --2- 172.21.15.19:0/3529335714 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0cc102960 0x7fe0cc19df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:14.889 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:14.887+0000 7fe0d0881700 1 -- 172.21.15.19:0/3529335714 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c80095f0 con 0x7fe0cc101fc0 2024-01-24T14:25:14.889 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:14.887+0000 7fe0d1082700 1 --2- 172.21.15.19:0/3529335714 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe0cc0feff0 0x7fe0cc1969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T14:25:14.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:14.887+0000 7fe0d0881700 1 --2- 172.21.15.19:0/3529335714 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe0cc101fc0 0x7fe0cc196ee0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe0c80059d0 tx=0x7fe0c8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:14.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:14.887+0000 7fe0c27fc700 1 -- 172.21.15.19:0/3529335714 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c801d070 con 0x7fe0cc101fc0 2024-01-24T14:25:14.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:14.887+0000 7fe0c27fc700 1 -- 172.21.15.19:0/3529335714 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe0c800fb90 con 0x7fe0cc101fc0 2024-01-24T14:25:14.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:14.887+0000 7fe0d32e6700 1 -- 172.21.15.19:0/3529335714 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0cc19ea70 con 0x7fe0cc101fc0 2024-01-24T14:25:14.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:14.887+0000 7fe0c27fc700 1 -- 172.21.15.19:0/3529335714 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c80173f0 con 0x7fe0cc101fc0 2024-01-24T14:25:14.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:14.887+0000 7fe0d32e6700 1 -- 172.21.15.19:0/3529335714 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0cc19ee90 con 0x7fe0cc101fc0 2024-01-24T14:25:14.891 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:14.888+0000 7fe0d32e6700 1 -- 172.21.15.19:0/3529335714 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0cc107d00 con 0x7fe0cc101fc0 2024-01-24T14:25:14.891 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:14.889+0000 7fe0c27fc700 1 -- 172.21.15.19:0/3529335714 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7fe0c800fd00 con 0x7fe0cc101fc0 2024-01-24T14:25:14.891 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:14.890+0000 7fe0c27fc700 1 --2- 172.21.15.19:0/3529335714 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fe0b80a0570 0x7fe0b80a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:14.891 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:14.890+0000 7fe0c27fc700 1 -- 172.21.15.19:0/3529335714 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fe0c80cd140 con 0x7fe0cc101fc0 2024-01-24T14:25:14.891 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:14.890+0000 7fe0d1082700 1 --2- 172.21.15.19:0/3529335714 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fe0b80a0570 0x7fe0b80a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:14.891 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:14.891+0000 7fe0d1082700 1 --2- 172.21.15.19:0/3529335714 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fe0b80a0570 0x7fe0b80a2a30 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc067ca0 tx=0x7fe0bc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:14.894 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:14.894+0000 7fe0c27fc700 1 -- 172.21.15.19:0/3529335714 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fe0c8096b80 con 0x7fe0cc101fc0 2024-01-24T14:25:15.109 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.109+0000 7fe0d32e6700 1 -- 172.21.15.19:0/3529335714 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0cc10c530 con 0x7fe0cc101fc0 2024-01-24T14:25:15.111 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.111+0000 7fe0c27fc700 1 -- 172.21.15.19:0/3529335714 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+23997 (secure 0 0 0) 0x7fe0c809a1a0 con 0x7fe0cc101fc0 2024-01-24T14:25:15.111 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:25:15.112 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":82,"fsid":"654b0c8e-bac3-11ee-95b1-87774f69a715","created":"2024-01-24T14:19:10.699369+0000","modified":"2024-01-24T14:25:07.960902+0000","last_up_change":"2024-01-24T14:24:30.032711+0000","last_in_change":"2024-01-24T14:24:13.776095+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T14:21:53.453429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-24T14:25:01.967538+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-24T14:25:02.864396+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-01-24T14:25:05.950591+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"81380ec6-8e19-4eb4-8158-8fafaed8d7e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":2456786579},{"type":"v1","addr":"172.21.15.19:6803","nonce":2456786579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":2456786579},{"type":"v1","addr":"172.21.15.19:6805","nonce":2456786579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":2456786579},{"type":"v1","addr":"172.21.15.19:6809","nonce":2456786579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":2456786579},{"type":"v1","addr":"172.21.15.19:6807","nonce":2456786579}]},"public_addr":"172.21.15.19:6803/2456786579","cluster_addr":"172.21.15.19:6805/2456786579","heartbeat_back_addr":"172.21.15.19:6809/2456786579","heartbeat_front_addr":"172.21.15.19:6807/2456786579","state":["exists","up"]},{"osd":1,"uuid":"687f5036-a6d2-4aa1-86e7-7570e51ef179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":2407548889},{"type":"v1","addr":"172.21.15.123:6801","nonce":2407548889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":2407548889},{"type":"v1","addr":"172.21.15.123:6803","nonce":2407548889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":2407548889},{"type":"v1","addr":"172.21.15.123:6807","nonce":2407548889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":2407548889},{"type":"v1","addr":"172.21.15.123:6805","nonce":2407548889}]},"public_addr":"172.21.15.123:6801/2407548889","cluster_addr":"172.21.15.123:6803/2407548889","heartbeat_back_addr":"172.21.15.123:6807/2407548889","heartbeat_front_addr":"172.21.15.123:6805/2407548889","state":["exists","up"]},{"osd":2,"uuid":"3e810904-371d-4ca8-b43c-434957a2aa12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":3632698637},{"type":"v1","addr":"172.21.15.194:6801","nonce":3632698637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":3632698637},{"type":"v1","addr":"172.21.15.194:6803","nonce":3632698637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":3632698637},{"type":"v1","addr":"172.21.15.194:6807","nonce":3632698637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":3632698637},{"type":"v1","addr":"172.21.15.194:6805","nonce":3632698637}]},"public_addr":"172.21.15.194:6801/3632698637","cluster_addr":"172.21.15.194:6803/3632698637","heartbeat_back_addr":"172.21.15.194:6807/3632698637","heartbeat_front_addr":"172.21.15.194:6805/3632698637","state":["exists","up"]},{"osd":3,"uuid":"3665d6e0-010a-40bd-b0c6-682af56a0cbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":3868219159},{"type":"v1","addr":"172.21.15.19:6811","nonce":3868219159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3868219159},{"type":"v1","addr":"172.21.15.19:6813","nonce":3868219159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":3868219159},{"type":"v1","addr":"172.21.15.19:6817","nonce":3868219159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":3868219159},{"type":"v1","addr":"172.21.15.19:6815","nonce":3868219159}]},"public_addr":"172.21.15.19:6811/3868219159","cluster_addr":"172.21.15.19:6813/3868219159","heartbeat_back_addr":"172.21.15.19:6817/3868219159","heartbeat_front_addr":"172.21.15.19:6815/3868219159","state":["exists","up"]},{"osd":4,"uuid":"029fb28f-9af5-4239-8ee5-5ac5d04cd34b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":3611579809},{"type":"v1","addr":"172.21.15.123:6809","nonce":3611579809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":3611579809},{"type":"v1","addr":"172.21.15.123:6811","nonce":3611579809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":3611579809},{"type":"v1","addr":"172.21.15.123:6815","nonce":3611579809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":3611579809},{"type":"v1","addr":"172.21.15.123:6813","nonce":3611579809}]},"public_addr":"172.21.15.123:6809/3611579809","cluster_addr":"172.21.15.123:6811/3611579809","heartbeat_back_addr":"172.21.15.123:6815/3611579809","heartbeat_front_addr":"172.21.15.123:6813/3611579809","state":["exists","up"]},{"osd":5,"uuid":"6e97f23b-f803-4fed-a0a2-7aee7de4970f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":2483361871},{"type":"v1","addr":"172.21.15.194:6809","nonce":2483361871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":2483361871},{"type":"v1","addr":"172.21.15.194:6811","nonce":2483361871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":2483361871},{"type":"v1","addr":"172.21.15.194:6815","nonce":2483361871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":2483361871},{"type":"v1","addr":"172.21.15.194:6813","nonce":2483361871}]},"public_addr":"172.21.15.194:6809/2483361871","cluster_addr":"172.21.15.194:6811/2483361871","heartbeat_back_addr":"172.21.15.194:6815/2483361871","heartbeat_front_addr":"172.21.15.194:6813/2483361871","state":["exists","up"]},{"osd":6,"uuid":"c8c8ca2e-4cc6-4861-94b0-6211e4936c25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":1290513386},{"type":"v1","addr":"172.21.15.19:6819","nonce":1290513386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":1290513386},{"type":"v1","addr":"172.21.15.19:6821","nonce":1290513386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":1290513386},{"type":"v1","addr":"172.21.15.19:6825","nonce":1290513386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":1290513386},{"type":"v1","addr":"172.21.15.19:6823","nonce":1290513386}]},"public_addr":"172.21.15.19:6819/1290513386","cluster_addr":"172.21.15.19:6821/1290513386","heartbeat_back_addr":"172.21.15.19:6825/1290513386","heartbeat_front_addr":"172.21.15.19:6823/1290513386","state":["exists","up"]},{"osd":7,"uuid":"9aaea7a8-7bc3-4e32-9328-20a6c30a6d8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1391966446},{"type":"v1","addr":"172.21.15.123:6817","nonce":1391966446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1391966446},{"type":"v1","addr":"172.21.15.123:6819","nonce":1391966446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1391966446},{"type":"v1","addr":"172.21.15.123:6823","nonce":1391966446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1391966446},{"type":"v1","addr":"172.21.15.123:6821","nonce":1391966446}]},"public_addr":"172.21.15.123:6817/1391966446","cluster_addr":"172.21.15.123:6819/1391966446","heartbeat_back_addr":"172.21.15.123:6823/1391966446","heartbeat_front_addr":"172.21.15.123:6821/1391966446","state":["exists","up"]},{"osd":8,"uuid":"e918ec1b-7608-4f73-9067-29082ea63999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":2260577139},{"type":"v1","addr":"172.21.15.194:6817","nonce":2260577139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":2260577139},{"type":"v1","addr":"172.21.15.194:6819","nonce":2260577139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":2260577139},{"type":"v1","addr":"172.21.15.194:6823","nonce":2260577139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":2260577139},{"type":"v1","addr":"172.21.15.194:6821","nonce":2260577139}]},"public_addr":"172.21.15.194:6817/2260577139","cluster_addr":"172.21.15.194:6819/2260577139","heartbeat_back_addr":"172.21.15.194:6823/2260577139","heartbeat_front_addr":"172.21.15.194:6821/2260577139","state":["exists","up"]},{"osd":9,"uuid":"0fd76722-3319-4828-ab68-108607a4bd0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":4004843075},{"type":"v1","addr":"172.21.15.19:6827","nonce":4004843075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":4004843075},{"type":"v1","addr":"172.21.15.19:6829","nonce":4004843075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":4004843075},{"type":"v1","addr":"172.21.15.19:6833","nonce":4004843075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":4004843075},{"type":"v1","addr":"172.21.15.19:6831","nonce":4004843075}]},"public_addr":"172.21.15.19:6827/4004843075","cluster_addr":"172.21.15.19:6829/4004843075","heartbeat_back_addr":"172.21.15.19:6833/4004843075","heartbeat_front_addr":"172.21.15.19:6831/4004843075","state":["exists","up"]},{"osd":10,"uuid":"624bb8f5-6d72-43d5-9986-1b48b8a564ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2591845068},{"type":"v1","addr":"172.21.15.123:6825","nonce":2591845068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":2591845068},{"type":"v1","addr":"172.21.15.123:6827","nonce":2591845068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2591845068},{"type":"v1","addr":"172.21.15.123:6831","nonce":2591845068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2591845068},{"type":"v1","addr":"172.21.15.123:6829","nonce":2591845068}]},"public_addr":"172.21.15.123:6825/2591845068","cluster_addr":"172.21.15.123:6827/2591845068","heartbeat_back_addr":"172.21.15.123:6831/2591845068","heartbeat_front_addr":"172.21.15.123:6829/2591845068","state":["exists","up"]},{"osd":11,"uuid":"219a55b3-b01e-471d-90da-1a3043bf3e34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":3985496260},{"type":"v1","addr":"172.21.15.194:6825","nonce":3985496260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":3985496260},{"type":"v1","addr":"172.21.15.194:6827","nonce":3985496260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":3985496260},{"type":"v1","addr":"172.21.15.194:6831","nonce":3985496260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":3985496260},{"type":"v1","addr":"172.21.15.194:6829","nonce":3985496260}]},"public_addr":"172.21.15.194:6825/3985496260","cluster_addr":"172.21.15.194:6827/3985496260","heartbeat_back_addr":"172.21.15.194:6831/3985496260","heartbeat_front_addr":"172.21.15.194:6829/3985496260","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:21:17.278347+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:21:34.469282+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:21:51.139919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:22:09.126458+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:22:26.186801+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:22:43.178997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:23:01.109570+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:23:18.592191+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:23:35.685046+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:23:53.574820+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:24:10.987531+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:24:28.231206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.19:0/1047252857":"2024-01-25T14:20:02.911218+0000","172.21.15.19:6801/3758061162":"2024-01-25T14:20:02.911218+0000","172.21.15.19:6800/3758061162":"2024-01-25T14:20:02.911218+0000","172.21.15.19:6801/2754937713":"2024-01-25T14:19:39.853212+0000","172.21.15.19:0/4062420470":"2024-01-25T14:19:39.853212+0000","172.21.15.19:6800/2754937713":"2024-01-25T14:19:39.853212+0000","172.21.15.19:0/822613692":"2024-01-25T14:20:02.911218+0000","172.21.15.19:0/3970666946":"2024-01-25T14:19:39.853212+0000","172.21.15.19:0/1116053938":"2024-01-25T14:19:39.853212+0000","172.21.15.19:0/1456770388":"2024-01-25T14:20:02.911218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T14:25:15.113 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.113+0000 7fe0d32e6700 1 -- 172.21.15.19:0/3529335714 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fe0b80a0570 msgr2=0x7fe0b80a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:15.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.113+0000 7fe0d32e6700 1 --2- 172.21.15.19:0/3529335714 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fe0b80a0570 0x7fe0b80a2a30 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc067ca0 tx=0x7fe0bc009260 comp rx=0 tx=0).stop 2024-01-24T14:25:15.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.113+0000 7fe0d32e6700 1 -- 172.21.15.19:0/3529335714 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe0cc101fc0 msgr2=0x7fe0cc196ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:15.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.113+0000 7fe0d32e6700 1 --2- 172.21.15.19:0/3529335714 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe0cc101fc0 0x7fe0cc196ee0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe0c80059d0 tx=0x7fe0c8004730 comp rx=0 tx=0).stop 2024-01-24T14:25:15.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.114+0000 7fe0d32e6700 1 -- 172.21.15.19:0/3529335714 shutdown_connections 2024-01-24T14:25:15.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.114+0000 7fe0d32e6700 1 --2- 172.21.15.19:0/3529335714 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe0cc102960 0x7fe0cc19df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:15.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.114+0000 7fe0d32e6700 1 --2- 172.21.15.19:0/3529335714 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe0cc0feff0 0x7fe0cc1969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:15.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.114+0000 7fe0d32e6700 1 --2- 172.21.15.19:0/3529335714 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fe0b80a0570 0x7fe0b80a2a30 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:15.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.114+0000 7fe0d32e6700 1 --2- 172.21.15.19:0/3529335714 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe0cc101fc0 0x7fe0cc196ee0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:15.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.114+0000 7fe0d32e6700 1 -- 172.21.15.19:0/3529335714 >> 172.21.15.19:0/3529335714 conn(0x7fe0cc0fa900 msgr2=0x7fe0cc0fcb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:15.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.114+0000 7fe0d32e6700 1 -- 172.21.15.19:0/3529335714 shutdown_connections 2024-01-24T14:25:15.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.114+0000 7fe0d32e6700 1 -- 172.21.15.19:0/3529335714 wait complete. 2024-01-24T14:25:15.127 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T14:21:53.453429+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T14:25:01.967538+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T14:25:02.864396+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-01-24T14:25:05.950591+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-24T14:25:15.128 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T14:25:15.270 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.267+0000 7efde00f5700 1 -- 172.21.15.19:0/2039088712 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efdd8101870 msgr2=0x7efdd8103c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:15.270 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.267+0000 7efde00f5700 1 --2- 172.21.15.19:0/2039088712 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efdd8101870 0x7efdd8103c60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7efdd4009990 tx=0x7efdd4009ca0 comp rx=0 tx=0).stop 2024-01-24T14:25:15.270 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.267+0000 7efde00f5700 1 -- 172.21.15.19:0/2039088712 shutdown_connections 2024-01-24T14:25:15.270 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.267+0000 7efde00f5700 1 --2- 172.21.15.19:0/2039088712 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efdd8101870 0x7efdd8103c60 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:15.270 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.267+0000 7efde00f5700 1 --2- 172.21.15.19:0/2039088712 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efdd8106ad0 0x7efdd8108ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:15.270 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.267+0000 7efde00f5700 1 --2- 172.21.15.19:0/2039088712 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efdd81041a0 0x7efdd8106590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:15.270 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.267+0000 7efde00f5700 1 -- 172.21.15.19:0/2039088712 >> 172.21.15.19:0/2039088712 conn(0x7efdd80fb1a0 msgr2=0x7efdd80fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:15.270 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.267+0000 7efde00f5700 1 -- 172.21.15.19:0/2039088712 shutdown_connections 2024-01-24T14:25:15.270 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.267+0000 7efde00f5700 1 -- 172.21.15.19:0/2039088712 wait complete. 2024-01-24T14:25:15.270 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.268+0000 7efde00f5700 1 Processor -- start 2024-01-24T14:25:15.270 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.268+0000 7efde00f5700 1 -- start start 2024-01-24T14:25:15.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.268+0000 7efde00f5700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efdd8101870 0x7efdd8196a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:15.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.268+0000 7efde00f5700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efdd81041a0 0x7efdd8196fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:15.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.268+0000 7efde00f5700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efdd8106ad0 0x7efdd819e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:15.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.268+0000 7efde00f5700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd819e5f0 con 0x7efdd8106ad0 2024-01-24T14:25:15.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.268+0000 7efde00f5700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd819e730 con 0x7efdd8101870 2024-01-24T14:25:15.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.268+0000 7efde00f5700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd819e8c0 con 0x7efdd81041a0 2024-01-24T14:25:15.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.268+0000 7efdde692700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efdd8106ad0 0x7efdd819e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:15.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.268+0000 7efdde692700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efdd8106ad0 0x7efdd819e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:57686/0 (socket says 172.21.15.19:57686) 2024-01-24T14:25:15.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.268+0000 7efdde692700 1 -- 172.21.15.19:0/3610743327 learned_addr learned my addr 172.21.15.19:0/3610743327 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:25:15.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.268+0000 7efddde91700 1 --2- 172.21.15.19:0/3610743327 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efdd8101870 0x7efdd8196a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:15.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.268+0000 7efddd690700 1 --2- 172.21.15.19:0/3610743327 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efdd81041a0 0x7efdd8196fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:15.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.268+0000 7efdde692700 1 -- 172.21.15.19:0/3610743327 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efdd81041a0 msgr2=0x7efdd8196fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:15.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.268+0000 7efdde692700 1 --2- 172.21.15.19:0/3610743327 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efdd81041a0 0x7efdd8196fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:15.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.268+0000 7efdde692700 1 -- 172.21.15.19:0/3610743327 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efdd8101870 msgr2=0x7efdd8196a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:15.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.268+0000 7efdde692700 1 --2- 172.21.15.19:0/3610743327 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efdd8101870 0x7efdd8196a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:15.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.268+0000 7efdde692700 1 -- 172.21.15.19:0/3610743327 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd40095f0 con 0x7efdd8106ad0 2024-01-24T14:25:15.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.269+0000 7efdde692700 1 --2- 172.21.15.19:0/3610743327 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efdd8106ad0 0x7efdd819e020 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7efdd0007ee0 tx=0x7efdd000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:15.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.269+0000 7efdceffd700 1 -- 172.21.15.19:0/3610743327 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd0017070 con 0x7efdd8106ad0 2024-01-24T14:25:15.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.269+0000 7efde00f5700 1 -- 172.21.15.19:0/3610743327 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdd819eba0 con 0x7efdd8106ad0 2024-01-24T14:25:15.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.269+0000 7efdceffd700 1 -- 172.21.15.19:0/3610743327 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efdd000f040 con 0x7efdd8106ad0 2024-01-24T14:25:15.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.269+0000 7efdceffd700 1 -- 172.21.15.19:0/3610743327 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd0005580 con 0x7efdd8106ad0 2024-01-24T14:25:15.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.269+0000 7efde00f5700 1 -- 172.21.15.19:0/3610743327 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdd819f0b0 con 0x7efdd8106ad0 2024-01-24T14:25:15.273 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.270+0000 7efde00f5700 1 -- 172.21.15.19:0/3610743327 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdd80fcd20 con 0x7efdd8106ad0 2024-01-24T14:25:15.273 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.271+0000 7efdceffd700 1 -- 172.21.15.19:0/3610743327 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7efdd000db20 con 0x7efdd8106ad0 2024-01-24T14:25:15.273 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.272+0000 7efdceffd700 1 --2- 172.21.15.19:0/3610743327 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7efdc40a0520 0x7efdc40a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:15.273 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.272+0000 7efddde91700 1 --2- 172.21.15.19:0/3610743327 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7efdc40a0520 0x7efdc40a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:15.273 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.272+0000 7efdceffd700 1 -- 172.21.15.19:0/3610743327 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8730+0+0 (secure 0 0 0) 0x7efdd00cb350 con 0x7efdd8106ad0 2024-01-24T14:25:15.273 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.272+0000 7efddde91700 1 --2- 172.21.15.19:0/3610743327 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7efdc40a0520 0x7efdc40a29e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7efdd400b5c0 tx=0x7efdd4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:15.276 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.276+0000 7efdceffd700 1 -- 172.21.15.19:0/3610743327 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7efdd0094ca0 con 0x7efdd8106ad0 2024-01-24T14:25:15.485 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.485+0000 7efde00f5700 1 -- 172.21.15.19:0/3610743327 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efdd8061950 con 0x7efdd8106ad0 2024-01-24T14:25:15.486 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.486+0000 7efdceffd700 1 -- 172.21.15.19:0/3610743327 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7efdd00982c0 con 0x7efdd8106ad0 2024-01-24T14:25:15.486 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 1 2024-01-24T14:25:15.489 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.488+0000 7efde00f5700 1 -- 172.21.15.19:0/3610743327 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7efdc40a0520 msgr2=0x7efdc40a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:15.489 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.488+0000 7efde00f5700 1 --2- 172.21.15.19:0/3610743327 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7efdc40a0520 0x7efdc40a29e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7efdd400b5c0 tx=0x7efdd4005fb0 comp rx=0 tx=0).stop 2024-01-24T14:25:15.489 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.489+0000 7efde00f5700 1 -- 172.21.15.19:0/3610743327 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efdd8106ad0 msgr2=0x7efdd819e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:15.489 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.489+0000 7efde00f5700 1 --2- 172.21.15.19:0/3610743327 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efdd8106ad0 0x7efdd819e020 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7efdd0007ee0 tx=0x7efdd000d3b0 comp rx=0 tx=0).stop 2024-01-24T14:25:15.489 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.489+0000 7efde00f5700 1 -- 172.21.15.19:0/3610743327 shutdown_connections 2024-01-24T14:25:15.489 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.489+0000 7efde00f5700 1 --2- 172.21.15.19:0/3610743327 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7efdc40a0520 0x7efdc40a29e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:15.489 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.489+0000 7efde00f5700 1 --2- 172.21.15.19:0/3610743327 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7efdd8106ad0 0x7efdd819e020 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:15.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.489+0000 7efde00f5700 1 --2- 172.21.15.19:0/3610743327 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efdd8101870 0x7efdd8196a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:15.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.489+0000 7efde00f5700 1 --2- 172.21.15.19:0/3610743327 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efdd81041a0 0x7efdd8196fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:15.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.489+0000 7efde00f5700 1 -- 172.21.15.19:0/3610743327 >> 172.21.15.19:0/3610743327 conn(0x7efdd80fb1a0 msgr2=0x7efdd81024e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:15.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.489+0000 7efde00f5700 1 -- 172.21.15.19:0/3610743327 shutdown_connections 2024-01-24T14:25:15.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.489+0000 7efde00f5700 1 -- 172.21.15.19:0/3610743327 wait complete. 2024-01-24T14:25:15.503 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T14:25:15.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.646+0000 7fbb5c771700 1 -- 172.21.15.19:0/2373917479 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbb54101850 msgr2=0x7fbb54103c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:15.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.646+0000 7fbb5c771700 1 --2- 172.21.15.19:0/2373917479 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbb54101850 0x7fbb54103c40 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fbb50009990 tx=0x7fbb50009ca0 comp rx=0 tx=0).stop 2024-01-24T14:25:15.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.646+0000 7fbb5c771700 1 -- 172.21.15.19:0/2373917479 shutdown_connections 2024-01-24T14:25:15.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.646+0000 7fbb5c771700 1 --2- 172.21.15.19:0/2373917479 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbb54101850 0x7fbb54103c40 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:15.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.646+0000 7fbb5c771700 1 --2- 172.21.15.19:0/2373917479 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbb54106ab0 0x7fbb54108ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:15.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.646+0000 7fbb5c771700 1 --2- 172.21.15.19:0/2373917479 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb54104180 0x7fbb54106570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:15.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.646+0000 7fbb5c771700 1 -- 172.21.15.19:0/2373917479 >> 172.21.15.19:0/2373917479 conn(0x7fbb540fb200 msgr2=0x7fbb540fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:15.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.646+0000 7fbb5c771700 1 -- 172.21.15.19:0/2373917479 shutdown_connections 2024-01-24T14:25:15.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.646+0000 7fbb5c771700 1 -- 172.21.15.19:0/2373917479 wait complete. 2024-01-24T14:25:15.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.647+0000 7fbb5c771700 1 Processor -- start 2024-01-24T14:25:15.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.647+0000 7fbb5c771700 1 -- start start 2024-01-24T14:25:15.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.647+0000 7fbb5c771700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbb54101850 0x7fbb54196a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:15.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.647+0000 7fbb5c771700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbb54104180 0x7fbb54196f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:15.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.647+0000 7fbb5c771700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb54106ab0 0x7fbb5419dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:15.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.647+0000 7fbb5c771700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb5419e5c0 con 0x7fbb54101850 2024-01-24T14:25:15.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.647+0000 7fbb5c771700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb5419e700 con 0x7fbb54106ab0 2024-01-24T14:25:15.650 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.647+0000 7fbb5c771700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb5419e890 con 0x7fbb54104180 2024-01-24T14:25:15.650 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.647+0000 7fbb5a50d700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbb54101850 0x7fbb54196a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:15.650 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.647+0000 7fbb5a50d700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbb54101850 0x7fbb54196a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:57700/0 (socket says 172.21.15.19:57700) 2024-01-24T14:25:15.650 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.647+0000 7fbb5a50d700 1 -- 172.21.15.19:0/60479165 learned_addr learned my addr 172.21.15.19:0/60479165 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:25:15.650 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.647+0000 7fbb5ad0e700 1 --2- 172.21.15.19:0/60479165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb54106ab0 0x7fbb5419dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:15.650 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.647+0000 7fbb59d0c700 1 --2- 172.21.15.19:0/60479165 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbb54104180 0x7fbb54196f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:15.650 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.647+0000 7fbb5a50d700 1 -- 172.21.15.19:0/60479165 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbb54104180 msgr2=0x7fbb54196f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:15.650 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.647+0000 7fbb5a50d700 1 --2- 172.21.15.19:0/60479165 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbb54104180 0x7fbb54196f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:15.650 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.647+0000 7fbb5a50d700 1 -- 172.21.15.19:0/60479165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb54106ab0 msgr2=0x7fbb5419dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:15.650 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.647+0000 7fbb5a50d700 1 --2- 172.21.15.19:0/60479165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb54106ab0 0x7fbb5419dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:15.650 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.647+0000 7fbb5a50d700 1 -- 172.21.15.19:0/60479165 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb500095f0 con 0x7fbb54101850 2024-01-24T14:25:15.650 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.647+0000 7fbb59d0c700 1 --2- 172.21.15.19:0/60479165 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbb54104180 0x7fbb54196f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-24T14:25:15.650 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.647+0000 7fbb5ad0e700 1 --2- 172.21.15.19:0/60479165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb54106ab0 0x7fbb5419dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T14:25:15.650 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.648+0000 7fbb5a50d700 1 --2- 172.21.15.19:0/60479165 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbb54101850 0x7fbb54196a40 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbb50004730 tx=0x7fbb50005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:15.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.648+0000 7fbb4b7fe700 1 -- 172.21.15.19:0/60479165 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb5001d070 con 0x7fbb54101850 2024-01-24T14:25:15.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.648+0000 7fbb4b7fe700 1 -- 172.21.15.19:0/60479165 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbb50004cd0 con 0x7fbb54101850 2024-01-24T14:25:15.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.648+0000 7fbb5c771700 1 -- 172.21.15.19:0/60479165 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb5419eb10 con 0x7fbb54101850 2024-01-24T14:25:15.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.648+0000 7fbb4b7fe700 1 -- 172.21.15.19:0/60479165 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb50005610 con 0x7fbb54101850 2024-01-24T14:25:15.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.648+0000 7fbb5c771700 1 -- 172.21.15.19:0/60479165 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb5419ef30 con 0x7fbb54101850 2024-01-24T14:25:15.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.649+0000 7fbb5c771700 1 -- 172.21.15.19:0/60479165 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb5410c5d0 con 0x7fbb54101850 2024-01-24T14:25:15.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.650+0000 7fbb4b7fe700 1 -- 172.21.15.19:0/60479165 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7fbb5000fc70 con 0x7fbb54101850 2024-01-24T14:25:15.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.651+0000 7fbb4b7fe700 1 --2- 172.21.15.19:0/60479165 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fbb400a05a0 0x7fbb400a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:15.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.651+0000 7fbb4b7fe700 1 -- 172.21.15.19:0/60479165 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fbb500cd3d0 con 0x7fbb54101850 2024-01-24T14:25:15.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.651+0000 7fbb59d0c700 1 --2- 172.21.15.19:0/60479165 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fbb400a05a0 0x7fbb400a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:15.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.651+0000 7fbb59d0c700 1 --2- 172.21.15.19:0/60479165 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fbb400a05a0 0x7fbb400a2a60 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fbb44009be0 tx=0x7fbb44009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:15.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.654+0000 7fbb4b7fe700 1 -- 172.21.15.19:0/60479165 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fbb50096e10 con 0x7fbb54101850 2024-01-24T14:25:15.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:15 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/3529335714' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T14:25:15.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:15 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/3529335714' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T14:25:15.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:15 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/3529335714' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T14:25:15.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.864+0000 7fbb5c771700 1 -- 172.21.15.19:0/60479165 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbb5419f5a0 con 0x7fbb54101850 2024-01-24T14:25:15.864 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.864+0000 7fbb4b7fe700 1 -- 172.21.15.19:0/60479165 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v82) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbb5009a430 con 0x7fbb54101850 2024-01-24T14:25:15.864 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 32 2024-01-24T14:25:15.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.866+0000 7fbb5c771700 1 -- 172.21.15.19:0/60479165 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fbb400a05a0 msgr2=0x7fbb400a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:15.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.866+0000 7fbb5c771700 1 --2- 172.21.15.19:0/60479165 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fbb400a05a0 0x7fbb400a2a60 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fbb44009be0 tx=0x7fbb44009260 comp rx=0 tx=0).stop 2024-01-24T14:25:15.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.866+0000 7fbb5c771700 1 -- 172.21.15.19:0/60479165 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbb54101850 msgr2=0x7fbb54196a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:15.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.866+0000 7fbb5c771700 1 --2- 172.21.15.19:0/60479165 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbb54101850 0x7fbb54196a40 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbb50004730 tx=0x7fbb50005ef0 comp rx=0 tx=0).stop 2024-01-24T14:25:15.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.866+0000 7fbb5c771700 1 -- 172.21.15.19:0/60479165 shutdown_connections 2024-01-24T14:25:15.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.866+0000 7fbb5c771700 1 --2- 172.21.15.19:0/60479165 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fbb400a05a0 0x7fbb400a2a60 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:15.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.866+0000 7fbb5c771700 1 --2- 172.21.15.19:0/60479165 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fbb54101850 0x7fbb54196a40 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:15.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.866+0000 7fbb5c771700 1 --2- 172.21.15.19:0/60479165 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbb54106ab0 0x7fbb5419dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:15.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.866+0000 7fbb5c771700 1 --2- 172.21.15.19:0/60479165 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbb54104180 0x7fbb54196f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:15.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.866+0000 7fbb5c771700 1 -- 172.21.15.19:0/60479165 >> 172.21.15.19:0/60479165 conn(0x7fbb540fb200 msgr2=0x7fbb54102cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:15.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.867+0000 7fbb5c771700 1 -- 172.21.15.19:0/60479165 shutdown_connections 2024-01-24T14:25:15.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:15.867+0000 7fbb5c771700 1 -- 172.21.15.19:0/60479165 wait complete. 2024-01-24T14:25:15.880 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T14:25:16.020 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.019+0000 7f9375779700 1 -- 172.21.15.19:0/2876083679 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9370100da0 msgr2=0x7f93701011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:16.020 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.019+0000 7f9375779700 1 --2- 172.21.15.19:0/2876083679 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9370100da0 0x7f93701011c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f93600099c0 tx=0x7f9360009cd0 comp rx=0 tx=0).stop 2024-01-24T14:25:16.020 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.019+0000 7f9375779700 1 -- 172.21.15.19:0/2876083679 shutdown_connections 2024-01-24T14:25:16.020 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.019+0000 7f9375779700 1 --2- 172.21.15.19:0/2876083679 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9370100da0 0x7f93701011c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:16.020 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.019+0000 7f9375779700 1 --2- 172.21.15.19:0/2876083679 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9370102930 0x7f93701091e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:16.021 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.019+0000 7f9375779700 1 --2- 172.21.15.19:0/2876083679 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9370101f90 0x7f93701023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:16.021 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.019+0000 7f9375779700 1 -- 172.21.15.19:0/2876083679 >> 172.21.15.19:0/2876083679 conn(0x7f93700fc310 msgr2=0x7f93700fe750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:16.021 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.019+0000 7f9375779700 1 -- 172.21.15.19:0/2876083679 shutdown_connections 2024-01-24T14:25:16.021 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.019+0000 7f9375779700 1 -- 172.21.15.19:0/2876083679 wait complete. 2024-01-24T14:25:16.021 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.020+0000 7f9375779700 1 Processor -- start 2024-01-24T14:25:16.021 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.020+0000 7f9375779700 1 -- start start 2024-01-24T14:25:16.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.020+0000 7f9375779700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9370100da0 0x7f9370196a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:16.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.020+0000 7f9375779700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9370101f90 0x7f9370196f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:16.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.020+0000 7f9375779700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9370102930 0x7f937019e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:16.024 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.020+0000 7f9375779700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f937019e5d0 con 0x7f9370101f90 2024-01-24T14:25:16.024 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.021+0000 7f9375779700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f937019e710 con 0x7f9370102930 2024-01-24T14:25:16.024 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.021+0000 7f9375779700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f937019e8a0 con 0x7f9370100da0 2024-01-24T14:25:16.024 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.021+0000 7f936e7fc700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9370101f90 0x7f9370196f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:16.024 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.021+0000 7f936e7fc700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9370101f90 0x7f9370196f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:57708/0 (socket says 172.21.15.19:57708) 2024-01-24T14:25:16.024 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.021+0000 7f936e7fc700 1 -- 172.21.15.19:0/1377469594 learned_addr learned my addr 172.21.15.19:0/1377469594 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:25:16.024 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.021+0000 7f936effd700 1 --2- 172.21.15.19:0/1377469594 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9370100da0 0x7f9370196a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:16.024 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.021+0000 7f936f7fe700 1 --2- 172.21.15.19:0/1377469594 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9370102930 0x7f937019e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:16.024 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.021+0000 7f936e7fc700 1 -- 172.21.15.19:0/1377469594 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9370100da0 msgr2=0x7f9370196a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:16.024 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.021+0000 7f936e7fc700 1 --2- 172.21.15.19:0/1377469594 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9370100da0 0x7f9370196a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:16.024 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.021+0000 7f936e7fc700 1 -- 172.21.15.19:0/1377469594 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9370102930 msgr2=0x7f937019e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:16.024 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.021+0000 7f936e7fc700 1 --2- 172.21.15.19:0/1377469594 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9370102930 0x7f937019e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:16.024 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.021+0000 7f936e7fc700 1 -- 172.21.15.19:0/1377469594 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93600095f0 con 0x7f9370101f90 2024-01-24T14:25:16.025 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.021+0000 7f936e7fc700 1 --2- 172.21.15.19:0/1377469594 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9370101f90 0x7f9370196f90 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f936400d950 tx=0x7f936400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:16.025 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.021+0000 7f9357fff700 1 -- 172.21.15.19:0/1377469594 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f936400f890 con 0x7f9370101f90 2024-01-24T14:25:16.025 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.021+0000 7f9357fff700 1 -- 172.21.15.19:0/1377469594 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9364010460 con 0x7f9370101f90 2024-01-24T14:25:16.025 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.021+0000 7f9375779700 1 -- 172.21.15.19:0/1377469594 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f937019eb80 con 0x7f9370101f90 2024-01-24T14:25:16.025 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.021+0000 7f9357fff700 1 -- 172.21.15.19:0/1377469594 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f936400e610 con 0x7f9370101f90 2024-01-24T14:25:16.025 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.021+0000 7f9375779700 1 -- 172.21.15.19:0/1377469594 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f937019f090 con 0x7f9370101f90 2024-01-24T14:25:16.025 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.023+0000 7f9375779700 1 -- 172.21.15.19:0/1377469594 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9370107d20 con 0x7f9370101f90 2024-01-24T14:25:16.025 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.024+0000 7f9357fff700 1 -- 172.21.15.19:0/1377469594 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f93640108d0 con 0x7f9370101f90 2024-01-24T14:25:16.026 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.024+0000 7f9357fff700 1 --2- 172.21.15.19:0/1377469594 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f93580a0520 0x7f93580a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:16.026 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.024+0000 7f9357fff700 1 -- 172.21.15.19:0/1377469594 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f93640cb920 con 0x7f9370101f90 2024-01-24T14:25:16.026 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.024+0000 7f936effd700 1 --2- 172.21.15.19:0/1377469594 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f93580a0520 0x7f93580a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:16.026 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.025+0000 7f936effd700 1 --2- 172.21.15.19:0/1377469594 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f93580a0520 0x7f93580a29e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f936000b580 tx=0x7f9360009f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:16.029 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.029+0000 7f9357fff700 1 -- 172.21.15.19:0/1377469594 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f93640952d0 con 0x7f9370101f90 2024-01-24T14:25:16.242 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.242+0000 7f9375779700 1 -- 172.21.15.19:0/1377469594 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f937002cc70 con 0x7f9370101f90 2024-01-24T14:25:16.242 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.242+0000 7f9357fff700 1 -- 172.21.15.19:0/1377469594 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v82) v1 ==== 100+0+11 (secure 0 0 0) 0x7f93640160a0 con 0x7f9370101f90 2024-01-24T14:25:16.243 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 64 2024-01-24T14:25:16.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.245+0000 7f9375779700 1 -- 172.21.15.19:0/1377469594 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f93580a0520 msgr2=0x7f93580a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:16.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.245+0000 7f9375779700 1 --2- 172.21.15.19:0/1377469594 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f93580a0520 0x7f93580a29e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f936000b580 tx=0x7f9360009f10 comp rx=0 tx=0).stop 2024-01-24T14:25:16.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.245+0000 7f9375779700 1 -- 172.21.15.19:0/1377469594 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9370101f90 msgr2=0x7f9370196f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:16.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.245+0000 7f9375779700 1 --2- 172.21.15.19:0/1377469594 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9370101f90 0x7f9370196f90 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f936400d950 tx=0x7f936400dc60 comp rx=0 tx=0).stop 2024-01-24T14:25:16.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.245+0000 7f9375779700 1 -- 172.21.15.19:0/1377469594 shutdown_connections 2024-01-24T14:25:16.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.245+0000 7f9375779700 1 --2- 172.21.15.19:0/1377469594 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9370102930 0x7f937019e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:16.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.245+0000 7f9375779700 1 --2- 172.21.15.19:0/1377469594 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9370100da0 0x7f9370196a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:16.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.245+0000 7f9375779700 1 --2- 172.21.15.19:0/1377469594 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f93580a0520 0x7f93580a29e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:16.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.245+0000 7f9375779700 1 --2- 172.21.15.19:0/1377469594 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9370101f90 0x7f9370196f90 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:16.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.245+0000 7f9375779700 1 -- 172.21.15.19:0/1377469594 >> 172.21.15.19:0/1377469594 conn(0x7f93700fc310 msgr2=0x7f93700fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:16.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.245+0000 7f9375779700 1 -- 172.21.15.19:0/1377469594 shutdown_connections 2024-01-24T14:25:16.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.245+0000 7f9375779700 1 -- 172.21.15.19:0/1377469594 wait complete. 2024-01-24T14:25:16.258 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-01-24T14:25:16.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.399+0000 7f9fdcbf6700 1 -- 172.21.15.19:0/1276909965 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9fd8101f40 msgr2=0x7f9fd81023a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:16.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.399+0000 7f9fdcbf6700 1 --2- 172.21.15.19:0/1276909965 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9fd8101f40 0x7f9fd81023a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc0099c0 tx=0x7f9fcc009cd0 comp rx=0 tx=0).stop 2024-01-24T14:25:16.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.399+0000 7f9fdcbf6700 1 -- 172.21.15.19:0/1276909965 shutdown_connections 2024-01-24T14:25:16.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.399+0000 7f9fdcbf6700 1 --2- 172.21.15.19:0/1276909965 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9fd81028e0 0x7f9fd8109190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:16.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.399+0000 7f9fdcbf6700 1 --2- 172.21.15.19:0/1276909965 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9fd8100d50 0x7f9fd8101170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:16.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.399+0000 7f9fdcbf6700 1 --2- 172.21.15.19:0/1276909965 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9fd8101f40 0x7f9fd81023a0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:16.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.399+0000 7f9fdcbf6700 1 -- 172.21.15.19:0/1276909965 >> 172.21.15.19:0/1276909965 conn(0x7f9fd80fc2c0 msgr2=0x7f9fd80fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:16.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.399+0000 7f9fdcbf6700 1 -- 172.21.15.19:0/1276909965 shutdown_connections 2024-01-24T14:25:16.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.400+0000 7f9fdcbf6700 1 -- 172.21.15.19:0/1276909965 wait complete. 2024-01-24T14:25:16.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.400+0000 7f9fdcbf6700 1 Processor -- start 2024-01-24T14:25:16.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.400+0000 7f9fdcbf6700 1 -- start start 2024-01-24T14:25:16.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.401+0000 7f9fdcbf6700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9fd8100d50 0x7f9fd8198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:16.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.401+0000 7f9fdcbf6700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9fd8101f40 0x7f9fd8199130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:16.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.401+0000 7f9fdcbf6700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9fd81028e0 0x7f9fd81a01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:16.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.401+0000 7f9fdcbf6700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd81a0770 con 0x7f9fd8100d50 2024-01-24T14:25:16.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.401+0000 7f9fdcbf6700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd81a08b0 con 0x7f9fd81028e0 2024-01-24T14:25:16.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.401+0000 7f9fdcbf6700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd81a0a40 con 0x7f9fd8101f40 2024-01-24T14:25:16.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.401+0000 7f9fd659c700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9fd8100d50 0x7f9fd8198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:16.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.401+0000 7f9fd659c700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9fd8100d50 0x7f9fd8198bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:57722/0 (socket says 172.21.15.19:57722) 2024-01-24T14:25:16.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.401+0000 7f9fd659c700 1 -- 172.21.15.19:0/1477010470 learned_addr learned my addr 172.21.15.19:0/1477010470 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:25:16.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.401+0000 7f9fd6d9d700 1 --2- 172.21.15.19:0/1477010470 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9fd81028e0 0x7f9fd81a01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:16.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.401+0000 7f9fd5d9b700 1 --2- 172.21.15.19:0/1477010470 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9fd8101f40 0x7f9fd8199130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:16.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.401+0000 7f9fd659c700 1 -- 172.21.15.19:0/1477010470 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9fd8101f40 msgr2=0x7f9fd8199130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:16.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.401+0000 7f9fd659c700 1 --2- 172.21.15.19:0/1477010470 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9fd8101f40 0x7f9fd8199130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:16.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.401+0000 7f9fd659c700 1 -- 172.21.15.19:0/1477010470 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9fd81028e0 msgr2=0x7f9fd81a01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:16.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.401+0000 7f9fd659c700 1 --2- 172.21.15.19:0/1477010470 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9fd81028e0 0x7f9fd81a01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:16.404 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.401+0000 7f9fd659c700 1 -- 172.21.15.19:0/1477010470 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fcc0095f0 con 0x7f9fd8100d50 2024-01-24T14:25:16.404 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.401+0000 7f9fd659c700 1 --2- 172.21.15.19:0/1477010470 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9fd8100d50 0x7f9fd8198bf0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9fc800e970 tx=0x7f9fc800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:16.404 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.401+0000 7f9fc77fe700 1 -- 172.21.15.19:0/1477010470 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc8009790 con 0x7f9fd8100d50 2024-01-24T14:25:16.404 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.401+0000 7f9fc77fe700 1 -- 172.21.15.19:0/1477010470 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9fc800c7e0 con 0x7f9fd8100d50 2024-01-24T14:25:16.404 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.401+0000 7f9fdcbf6700 1 -- 172.21.15.19:0/1477010470 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fd81a0d20 con 0x7f9fd8100d50 2024-01-24T14:25:16.404 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.401+0000 7f9fc77fe700 1 -- 172.21.15.19:0/1477010470 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc8003c70 con 0x7f9fd8100d50 2024-01-24T14:25:16.404 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.402+0000 7f9fdcbf6700 1 -- 172.21.15.19:0/1477010470 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fd81a11d0 con 0x7f9fd8100d50 2024-01-24T14:25:16.404 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.402+0000 7f9fdcbf6700 1 -- 172.21.15.19:0/1477010470 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fd810c5d0 con 0x7f9fd8100d50 2024-01-24T14:25:16.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.404+0000 7f9fc77fe700 1 -- 172.21.15.19:0/1477010470 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f9fc800cc50 con 0x7f9fd8100d50 2024-01-24T14:25:16.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.404+0000 7f9fc77fe700 1 --2- 172.21.15.19:0/1477010470 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f9fc00a05a0 0x7f9fc00a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:16.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.404+0000 7f9fc77fe700 1 -- 172.21.15.19:0/1477010470 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f9fc8014070 con 0x7f9fd8100d50 2024-01-24T14:25:16.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.404+0000 7f9fd5d9b700 1 --2- 172.21.15.19:0/1477010470 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f9fc00a05a0 0x7f9fc00a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:16.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.405+0000 7f9fd5d9b700 1 --2- 172.21.15.19:0/1477010470 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f9fc00a05a0 0x7f9fc00a2a60 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc00b580 tx=0x7f9fcc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:16.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.408+0000 7f9fc77fe700 1 -- 172.21.15.19:0/1477010470 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f9fc80957e0 con 0x7f9fd8100d50 2024-01-24T14:25:16.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.618+0000 7f9fdcbf6700 1 -- 172.21.15.19:0/1477010470 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f9fd81a1840 con 0x7f9fd8100d50 2024-01-24T14:25:16.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.619+0000 7f9fc77fe700 1 -- 172.21.15.19:0/1477010470 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v82) v1 ==== 103+0+11 (secure 0 0 0) 0x7f9fc8003dd0 con 0x7f9fd8100d50 2024-01-24T14:25:16.619 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 32 2024-01-24T14:25:16.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.621+0000 7f9fdcbf6700 1 -- 172.21.15.19:0/1477010470 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f9fc00a05a0 msgr2=0x7f9fc00a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:16.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.621+0000 7f9fdcbf6700 1 --2- 172.21.15.19:0/1477010470 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f9fc00a05a0 0x7f9fc00a2a60 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc00b580 tx=0x7f9fcc005fb0 comp rx=0 tx=0).stop 2024-01-24T14:25:16.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.621+0000 7f9fdcbf6700 1 -- 172.21.15.19:0/1477010470 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9fd8100d50 msgr2=0x7f9fd8198bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:16.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.621+0000 7f9fdcbf6700 1 --2- 172.21.15.19:0/1477010470 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9fd8100d50 0x7f9fd8198bf0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9fc800e970 tx=0x7f9fc800ec80 comp rx=0 tx=0).stop 2024-01-24T14:25:16.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.621+0000 7f9fdcbf6700 1 -- 172.21.15.19:0/1477010470 shutdown_connections 2024-01-24T14:25:16.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.621+0000 7f9fdcbf6700 1 --2- 172.21.15.19:0/1477010470 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f9fc00a05a0 0x7f9fc00a2a60 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:16.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.621+0000 7f9fdcbf6700 1 --2- 172.21.15.19:0/1477010470 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9fd8100d50 0x7f9fd8198bf0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:16.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.621+0000 7f9fdcbf6700 1 --2- 172.21.15.19:0/1477010470 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9fd81028e0 0x7f9fd81a01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:16.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.621+0000 7f9fdcbf6700 1 --2- 172.21.15.19:0/1477010470 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9fd8101f40 0x7f9fd8199130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:16.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.621+0000 7f9fdcbf6700 1 -- 172.21.15.19:0/1477010470 >> 172.21.15.19:0/1477010470 conn(0x7f9fd80fc2c0 msgr2=0x7f9fd80fe560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:16.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.621+0000 7f9fdcbf6700 1 -- 172.21.15.19:0/1477010470 shutdown_connections 2024-01-24T14:25:16.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.621+0000 7f9fdcbf6700 1 -- 172.21.15.19:0/1477010470 wait complete. 2024-01-24T14:25:16.634 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:16 smithi019 ceph-mon[40663]: pgmap v228: 129 pgs: 129 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-01-24T14:25:16.634 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:16 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/3610743327' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-01-24T14:25:16.634 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:16 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/60479165' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-01-24T14:25:16.634 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:16 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/1377469594' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-01-24T14:25:16.635 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-24T14:25:16.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.773+0000 7f41e3e41700 1 -- 172.21.15.19:0/1651882391 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f41dc100e20 msgr2=0x7f41dc101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:16.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.773+0000 7f41e3e41700 1 --2- 172.21.15.19:0/1651882391 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f41dc100e20 0x7f41dc101280 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f41cc0099c0 tx=0x7f41cc009cd0 comp rx=0 tx=0).stop 2024-01-24T14:25:16.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.773+0000 7f41e3e41700 1 -- 172.21.15.19:0/1651882391 shutdown_connections 2024-01-24T14:25:16.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.773+0000 7f41e3e41700 1 --2- 172.21.15.19:0/1651882391 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41dc1017c0 0x7f41dc108070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:16.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.773+0000 7f41e3e41700 1 --2- 172.21.15.19:0/1651882391 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41dc0ffc30 0x7f41dc100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:16.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.773+0000 7f41e3e41700 1 --2- 172.21.15.19:0/1651882391 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f41dc100e20 0x7f41dc101280 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:16.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.773+0000 7f41e3e41700 1 -- 172.21.15.19:0/1651882391 >> 172.21.15.19:0/1651882391 conn(0x7f41dc0fb1a0 msgr2=0x7f41dc0fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:16.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.773+0000 7f41e3e41700 1 -- 172.21.15.19:0/1651882391 shutdown_connections 2024-01-24T14:25:16.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.773+0000 7f41e3e41700 1 -- 172.21.15.19:0/1651882391 wait complete. 2024-01-24T14:25:16.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.774+0000 7f41e3e41700 1 Processor -- start 2024-01-24T14:25:16.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.774+0000 7f41e3e41700 1 -- start start 2024-01-24T14:25:16.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.774+0000 7f41e3e41700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f41dc0ffc30 0x7f41dc196a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:16.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.774+0000 7f41e3e41700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41dc100e20 0x7f41dc196f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:16.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.774+0000 7f41e3e41700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41dc1017c0 0x7f41dc19dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:16.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.774+0000 7f41e3e41700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41dc19e5a0 con 0x7f41dc0ffc30 2024-01-24T14:25:16.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.774+0000 7f41e3e41700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41dc19e6e0 con 0x7f41dc100e20 2024-01-24T14:25:16.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.774+0000 7f41e3e41700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41dc19e870 con 0x7f41dc1017c0 2024-01-24T14:25:16.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.774+0000 7f41e1bdd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f41dc0ffc30 0x7f41dc196a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:16.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.774+0000 7f41e1bdd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f41dc0ffc30 0x7f41dc196a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:57748/0 (socket says 172.21.15.19:57748) 2024-01-24T14:25:16.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.774+0000 7f41e13dc700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41dc100e20 0x7f41dc196f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:16.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.774+0000 7f41e1bdd700 1 -- 172.21.15.19:0/2127339402 learned_addr learned my addr 172.21.15.19:0/2127339402 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:25:16.776 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.774+0000 7f41e23de700 1 --2- 172.21.15.19:0/2127339402 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41dc1017c0 0x7f41dc19dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:16.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.775+0000 7f41e1bdd700 1 -- 172.21.15.19:0/2127339402 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41dc1017c0 msgr2=0x7f41dc19dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:16.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.775+0000 7f41e1bdd700 1 --2- 172.21.15.19:0/2127339402 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41dc1017c0 0x7f41dc19dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:16.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.775+0000 7f41e1bdd700 1 -- 172.21.15.19:0/2127339402 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41dc100e20 msgr2=0x7f41dc196f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:16.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.775+0000 7f41e1bdd700 1 --2- 172.21.15.19:0/2127339402 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41dc100e20 0x7f41dc196f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:16.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.775+0000 7f41e1bdd700 1 -- 172.21.15.19:0/2127339402 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41cc0095f0 con 0x7f41dc0ffc30 2024-01-24T14:25:16.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.775+0000 7f41e13dc700 1 --2- 172.21.15.19:0/2127339402 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41dc100e20 0x7f41dc196f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T14:25:16.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.775+0000 7f41e1bdd700 1 --2- 172.21.15.19:0/2127339402 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f41dc0ffc30 0x7f41dc196a20 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f41d800d980 tx=0x7f41d800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:16.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.775+0000 7f41d2ffd700 1 -- 172.21.15.19:0/2127339402 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d800b950 con 0x7f41dc0ffc30 2024-01-24T14:25:16.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.775+0000 7f41d2ffd700 1 -- 172.21.15.19:0/2127339402 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f41d8007620 con 0x7f41dc0ffc30 2024-01-24T14:25:16.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.775+0000 7f41e3e41700 1 -- 172.21.15.19:0/2127339402 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41dc19eb50 con 0x7f41dc0ffc30 2024-01-24T14:25:16.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.775+0000 7f41d2ffd700 1 -- 172.21.15.19:0/2127339402 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d800f3c0 con 0x7f41dc0ffc30 2024-01-24T14:25:16.777 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.775+0000 7f41e3e41700 1 -- 172.21.15.19:0/2127339402 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41dc19f0b0 con 0x7f41dc0ffc30 2024-01-24T14:25:16.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.776+0000 7f41e3e41700 1 -- 172.21.15.19:0/2127339402 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41dc106bb0 con 0x7f41dc0ffc30 2024-01-24T14:25:16.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.777+0000 7f41d2ffd700 1 -- 172.21.15.19:0/2127339402 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f41d8010640 con 0x7f41dc0ffc30 2024-01-24T14:25:16.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.778+0000 7f41d2ffd700 1 --2- 172.21.15.19:0/2127339402 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f41c80a05a0 0x7f41c80a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:16.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.778+0000 7f41d2ffd700 1 -- 172.21.15.19:0/2127339402 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f41d80cc290 con 0x7f41dc0ffc30 2024-01-24T14:25:16.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.778+0000 7f41e13dc700 1 --2- 172.21.15.19:0/2127339402 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f41c80a05a0 0x7f41c80a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:16.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.778+0000 7f41e13dc700 1 --2- 172.21.15.19:0/2127339402 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f41c80a05a0 0x7f41c80a2a60 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f41cc00b580 tx=0x7f41cc005bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:16.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:16.781+0000 7f41d2ffd700 1 -- 172.21.15.19:0/2127339402 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f41d8095c40 con 0x7f41dc0ffc30 2024-01-24T14:25:16.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:16 smithi123 ceph-mon[44966]: pgmap v228: 129 pgs: 129 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-01-24T14:25:16.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:16 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/3610743327' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-01-24T14:25:16.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:16 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/60479165' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-01-24T14:25:16.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:16 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/1377469594' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-01-24T14:25:16.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:16 smithi194 ceph-mon[45450]: pgmap v228: 129 pgs: 129 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-01-24T14:25:16.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:16 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/3610743327' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-01-24T14:25:16.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:16 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/60479165' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-01-24T14:25:16.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:16 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/1377469594' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-01-24T14:25:17.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.009+0000 7f41e3e41700 1 -- 172.21.15.19:0/2127339402 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f41dc02cc70 con 0x7f41dc0ffc30 2024-01-24T14:25:17.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.009+0000 7f41d2ffd700 1 -- 172.21.15.19:0/2127339402 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v82) v1 ==== 81+0+591 (secure 0 0 0) 0x7f41d8016080 con 0x7f41dc0ffc30 2024-01-24T14:25:17.011 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:0/1047252857 2024-01-25T14:20:02.911218+0000 2024-01-24T14:25:17.011 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:6801/3758061162 2024-01-25T14:20:02.911218+0000 2024-01-24T14:25:17.011 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:6800/3758061162 2024-01-25T14:20:02.911218+0000 2024-01-24T14:25:17.011 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:6801/2754937713 2024-01-25T14:19:39.853212+0000 2024-01-24T14:25:17.012 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:0/4062420470 2024-01-25T14:19:39.853212+0000 2024-01-24T14:25:17.012 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:6800/2754937713 2024-01-25T14:19:39.853212+0000 2024-01-24T14:25:17.012 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:0/822613692 2024-01-25T14:20:02.911218+0000 2024-01-24T14:25:17.012 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:0/3970666946 2024-01-25T14:19:39.853212+0000 2024-01-24T14:25:17.012 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:0/1116053938 2024-01-25T14:19:39.853212+0000 2024-01-24T14:25:17.012 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:0/1456770388 2024-01-25T14:20:02.911218+0000 2024-01-24T14:25:17.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.012+0000 7f41e3e41700 1 -- 172.21.15.19:0/2127339402 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f41c80a05a0 msgr2=0x7f41c80a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:17.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.012+0000 7f41e3e41700 1 --2- 172.21.15.19:0/2127339402 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f41c80a05a0 0x7f41c80a2a60 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f41cc00b580 tx=0x7f41cc005bc0 comp rx=0 tx=0).stop 2024-01-24T14:25:17.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.012+0000 7f41e3e41700 1 -- 172.21.15.19:0/2127339402 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f41dc0ffc30 msgr2=0x7f41dc196a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:17.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.012+0000 7f41e3e41700 1 --2- 172.21.15.19:0/2127339402 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f41dc0ffc30 0x7f41dc196a20 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f41d800d980 tx=0x7f41d800dc90 comp rx=0 tx=0).stop 2024-01-24T14:25:17.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.012+0000 7f41e3e41700 1 -- 172.21.15.19:0/2127339402 shutdown_connections 2024-01-24T14:25:17.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.012+0000 7f41e3e41700 1 --2- 172.21.15.19:0/2127339402 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f41c80a05a0 0x7f41c80a2a60 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:17.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.012+0000 7f41e3e41700 1 --2- 172.21.15.19:0/2127339402 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f41dc0ffc30 0x7f41dc196a20 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:17.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.012+0000 7f41e3e41700 1 --2- 172.21.15.19:0/2127339402 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41dc1017c0 0x7f41dc19dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:17.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.012+0000 7f41e3e41700 1 --2- 172.21.15.19:0/2127339402 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41dc100e20 0x7f41dc196f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:17.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.012+0000 7f41e3e41700 1 -- 172.21.15.19:0/2127339402 >> 172.21.15.19:0/2127339402 conn(0x7f41dc0fb1a0 msgr2=0x7f41dc0fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:17.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.012+0000 7f41e3e41700 1 -- 172.21.15.19:0/2127339402 shutdown_connections 2024-01-24T14:25:17.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.012+0000 7f41e3e41700 1 -- 172.21.15.19:0/2127339402 wait complete. 2024-01-24T14:25:17.017 INFO:teuthology.orchestra.run.smithi019.stderr:listed 10 entries 2024-01-24T14:25:17.027 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-01-24T14:25:17.028 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-24T14:25:17.048 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T14:25:17.232 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.230+0000 7fd601aec700 1 -- 172.21.15.19:0/1230877171 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd5fc104170 msgr2=0x7fd5fc106560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:17.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.230+0000 7fd601aec700 1 --2- 172.21.15.19:0/1230877171 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd5fc104170 0x7fd5fc106560 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd5f0009990 tx=0x7fd5f0009ca0 comp rx=0 tx=0).stop 2024-01-24T14:25:17.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.231+0000 7fd601aec700 1 -- 172.21.15.19:0/1230877171 shutdown_connections 2024-01-24T14:25:17.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.231+0000 7fd601aec700 1 --2- 172.21.15.19:0/1230877171 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd5fc106aa0 0x7fd5fc108e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:17.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.231+0000 7fd601aec700 1 --2- 172.21.15.19:0/1230877171 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5fc101840 0x7fd5fc103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:17.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.231+0000 7fd601aec700 1 --2- 172.21.15.19:0/1230877171 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd5fc104170 0x7fd5fc106560 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:17.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.231+0000 7fd601aec700 1 -- 172.21.15.19:0/1230877171 >> 172.21.15.19:0/1230877171 conn(0x7fd5fc0fb170 msgr2=0x7fd5fc0fd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:17.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.231+0000 7fd601aec700 1 -- 172.21.15.19:0/1230877171 shutdown_connections 2024-01-24T14:25:17.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.231+0000 7fd601aec700 1 -- 172.21.15.19:0/1230877171 wait complete. 2024-01-24T14:25:17.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.231+0000 7fd601aec700 1 Processor -- start 2024-01-24T14:25:17.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.231+0000 7fd601aec700 1 -- start start 2024-01-24T14:25:17.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.232+0000 7fd601aec700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd5fc101840 0x7fd5fc1969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:17.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.232+0000 7fd601aec700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5fc104170 0x7fd5fc196ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:17.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.232+0000 7fd601aec700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd5fc106aa0 0x7fd5fc19df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:17.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.232+0000 7fd601aec700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5fc19e520 con 0x7fd5fc101840 2024-01-24T14:25:17.234 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.232+0000 7fd601aec700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5fc19e660 con 0x7fd5fc104170 2024-01-24T14:25:17.234 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.232+0000 7fd601aec700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5fc19e7f0 con 0x7fd5fc106aa0 2024-01-24T14:25:17.235 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.232+0000 7fd5fb7fe700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd5fc101840 0x7fd5fc1969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:17.235 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.232+0000 7fd5fb7fe700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd5fc101840 0x7fd5fc1969a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:57754/0 (socket says 172.21.15.19:57754) 2024-01-24T14:25:17.235 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.232+0000 7fd5fb7fe700 1 -- 172.21.15.19:0/3908232960 learned_addr learned my addr 172.21.15.19:0/3908232960 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:25:17.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.232+0000 7fd5faffd700 1 --2- 172.21.15.19:0/3908232960 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5fc104170 0x7fd5fc196ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:17.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.232+0000 7fd5fbfff700 1 --2- 172.21.15.19:0/3908232960 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd5fc106aa0 0x7fd5fc19df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:17.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.232+0000 7fd5faffd700 1 -- 172.21.15.19:0/3908232960 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd5fc106aa0 msgr2=0x7fd5fc19df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:17.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.232+0000 7fd5faffd700 1 --2- 172.21.15.19:0/3908232960 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd5fc106aa0 0x7fd5fc19df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:17.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.232+0000 7fd5faffd700 1 -- 172.21.15.19:0/3908232960 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd5fc101840 msgr2=0x7fd5fc1969a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:17.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.232+0000 7fd5faffd700 1 --2- 172.21.15.19:0/3908232960 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd5fc101840 0x7fd5fc1969a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:17.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.232+0000 7fd5faffd700 1 -- 172.21.15.19:0/3908232960 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5f00095f0 con 0x7fd5fc104170 2024-01-24T14:25:17.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.232+0000 7fd5fb7fe700 1 --2- 172.21.15.19:0/3908232960 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd5fc101840 0x7fd5fc1969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T14:25:17.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.232+0000 7fd5faffd700 1 --2- 172.21.15.19:0/3908232960 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5fc104170 0x7fd5fc196ee0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd5f0000c00 tx=0x7fd5f0005720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:17.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.233+0000 7fd5f8ff9700 1 -- 172.21.15.19:0/3908232960 <== mon.2 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f001d070 con 0x7fd5fc104170 2024-01-24T14:25:17.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.233+0000 7fd601aec700 1 -- 172.21.15.19:0/3908232960 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5fc19ea70 con 0x7fd5fc104170 2024-01-24T14:25:17.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.233+0000 7fd5f8ff9700 1 -- 172.21.15.19:0/3908232960 <== mon.2 v2:172.21.15.123:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd5f0022470 con 0x7fd5fc104170 2024-01-24T14:25:17.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.233+0000 7fd5f8ff9700 1 -- 172.21.15.19:0/3908232960 <== mon.2 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f0017650 con 0x7fd5fc104170 2024-01-24T14:25:17.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.233+0000 7fd601aec700 1 -- 172.21.15.19:0/3908232960 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5fc19eef0 con 0x7fd5fc104170 2024-01-24T14:25:17.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.234+0000 7fd601aec700 1 -- 172.21.15.19:0/3908232960 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5fc10c500 con 0x7fd5fc104170 2024-01-24T14:25:17.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.235+0000 7fd5f8ff9700 1 -- 172.21.15.19:0/3908232960 <== mon.2 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7fd5f0022870 con 0x7fd5fc104170 2024-01-24T14:25:17.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.236+0000 7fd5f8ff9700 1 --2- 172.21.15.19:0/3908232960 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fd5e40a0570 0x7fd5e40a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:17.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.236+0000 7fd5fb7fe700 1 --2- 172.21.15.19:0/3908232960 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fd5e40a0570 0x7fd5e40a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:17.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.236+0000 7fd5f8ff9700 1 -- 172.21.15.19:0/3908232960 <== mon.2 v2:172.21.15.123:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fd5f00cd720 con 0x7fd5fc104170 2024-01-24T14:25:17.237 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.236+0000 7fd5fb7fe700 1 --2- 172.21.15.19:0/3908232960 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fd5e40a0570 0x7fd5e40a2a30 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec011550 tx=0x7fd5ec009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:17.240 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.239+0000 7fd5f8ff9700 1 -- 172.21.15.19:0/3908232960 <== mon.2 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fd5f0097160 con 0x7fd5fc104170 2024-01-24T14:25:17.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.455+0000 7fd601aec700 1 -- 172.21.15.19:0/3908232960 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5fc19f560 con 0x7fd5fc104170 2024-01-24T14:25:17.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.457+0000 7fd5f8ff9700 1 -- 172.21.15.19:0/3908232960 <== mon.2 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+23997 (secure 0 0 0) 0x7fd5f009a780 con 0x7fd5fc104170 2024-01-24T14:25:17.458 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:25:17.459 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":82,"fsid":"654b0c8e-bac3-11ee-95b1-87774f69a715","created":"2024-01-24T14:19:10.699369+0000","modified":"2024-01-24T14:25:07.960902+0000","last_up_change":"2024-01-24T14:24:30.032711+0000","last_in_change":"2024-01-24T14:24:13.776095+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T14:21:53.453429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-24T14:25:01.967538+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-24T14:25:02.864396+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-01-24T14:25:05.950591+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"81380ec6-8e19-4eb4-8158-8fafaed8d7e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":2456786579},{"type":"v1","addr":"172.21.15.19:6803","nonce":2456786579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":2456786579},{"type":"v1","addr":"172.21.15.19:6805","nonce":2456786579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":2456786579},{"type":"v1","addr":"172.21.15.19:6809","nonce":2456786579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":2456786579},{"type":"v1","addr":"172.21.15.19:6807","nonce":2456786579}]},"public_addr":"172.21.15.19:6803/2456786579","cluster_addr":"172.21.15.19:6805/2456786579","heartbeat_back_addr":"172.21.15.19:6809/2456786579","heartbeat_front_addr":"172.21.15.19:6807/2456786579","state":["exists","up"]},{"osd":1,"uuid":"687f5036-a6d2-4aa1-86e7-7570e51ef179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":2407548889},{"type":"v1","addr":"172.21.15.123:6801","nonce":2407548889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":2407548889},{"type":"v1","addr":"172.21.15.123:6803","nonce":2407548889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":2407548889},{"type":"v1","addr":"172.21.15.123:6807","nonce":2407548889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":2407548889},{"type":"v1","addr":"172.21.15.123:6805","nonce":2407548889}]},"public_addr":"172.21.15.123:6801/2407548889","cluster_addr":"172.21.15.123:6803/2407548889","heartbeat_back_addr":"172.21.15.123:6807/2407548889","heartbeat_front_addr":"172.21.15.123:6805/2407548889","state":["exists","up"]},{"osd":2,"uuid":"3e810904-371d-4ca8-b43c-434957a2aa12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":3632698637},{"type":"v1","addr":"172.21.15.194:6801","nonce":3632698637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":3632698637},{"type":"v1","addr":"172.21.15.194:6803","nonce":3632698637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":3632698637},{"type":"v1","addr":"172.21.15.194:6807","nonce":3632698637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":3632698637},{"type":"v1","addr":"172.21.15.194:6805","nonce":3632698637}]},"public_addr":"172.21.15.194:6801/3632698637","cluster_addr":"172.21.15.194:6803/3632698637","heartbeat_back_addr":"172.21.15.194:6807/3632698637","heartbeat_front_addr":"172.21.15.194:6805/3632698637","state":["exists","up"]},{"osd":3,"uuid":"3665d6e0-010a-40bd-b0c6-682af56a0cbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":3868219159},{"type":"v1","addr":"172.21.15.19:6811","nonce":3868219159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3868219159},{"type":"v1","addr":"172.21.15.19:6813","nonce":3868219159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":3868219159},{"type":"v1","addr":"172.21.15.19:6817","nonce":3868219159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":3868219159},{"type":"v1","addr":"172.21.15.19:6815","nonce":3868219159}]},"public_addr":"172.21.15.19:6811/3868219159","cluster_addr":"172.21.15.19:6813/3868219159","heartbeat_back_addr":"172.21.15.19:6817/3868219159","heartbeat_front_addr":"172.21.15.19:6815/3868219159","state":["exists","up"]},{"osd":4,"uuid":"029fb28f-9af5-4239-8ee5-5ac5d04cd34b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":3611579809},{"type":"v1","addr":"172.21.15.123:6809","nonce":3611579809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":3611579809},{"type":"v1","addr":"172.21.15.123:6811","nonce":3611579809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":3611579809},{"type":"v1","addr":"172.21.15.123:6815","nonce":3611579809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":3611579809},{"type":"v1","addr":"172.21.15.123:6813","nonce":3611579809}]},"public_addr":"172.21.15.123:6809/3611579809","cluster_addr":"172.21.15.123:6811/3611579809","heartbeat_back_addr":"172.21.15.123:6815/3611579809","heartbeat_front_addr":"172.21.15.123:6813/3611579809","state":["exists","up"]},{"osd":5,"uuid":"6e97f23b-f803-4fed-a0a2-7aee7de4970f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":2483361871},{"type":"v1","addr":"172.21.15.194:6809","nonce":2483361871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":2483361871},{"type":"v1","addr":"172.21.15.194:6811","nonce":2483361871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":2483361871},{"type":"v1","addr":"172.21.15.194:6815","nonce":2483361871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":2483361871},{"type":"v1","addr":"172.21.15.194:6813","nonce":2483361871}]},"public_addr":"172.21.15.194:6809/2483361871","cluster_addr":"172.21.15.194:6811/2483361871","heartbeat_back_addr":"172.21.15.194:6815/2483361871","heartbeat_front_addr":"172.21.15.194:6813/2483361871","state":["exists","up"]},{"osd":6,"uuid":"c8c8ca2e-4cc6-4861-94b0-6211e4936c25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":1290513386},{"type":"v1","addr":"172.21.15.19:6819","nonce":1290513386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":1290513386},{"type":"v1","addr":"172.21.15.19:6821","nonce":1290513386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":1290513386},{"type":"v1","addr":"172.21.15.19:6825","nonce":1290513386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":1290513386},{"type":"v1","addr":"172.21.15.19:6823","nonce":1290513386}]},"public_addr":"172.21.15.19:6819/1290513386","cluster_addr":"172.21.15.19:6821/1290513386","heartbeat_back_addr":"172.21.15.19:6825/1290513386","heartbeat_front_addr":"172.21.15.19:6823/1290513386","state":["exists","up"]},{"osd":7,"uuid":"9aaea7a8-7bc3-4e32-9328-20a6c30a6d8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1391966446},{"type":"v1","addr":"172.21.15.123:6817","nonce":1391966446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1391966446},{"type":"v1","addr":"172.21.15.123:6819","nonce":1391966446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1391966446},{"type":"v1","addr":"172.21.15.123:6823","nonce":1391966446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1391966446},{"type":"v1","addr":"172.21.15.123:6821","nonce":1391966446}]},"public_addr":"172.21.15.123:6817/1391966446","cluster_addr":"172.21.15.123:6819/1391966446","heartbeat_back_addr":"172.21.15.123:6823/1391966446","heartbeat_front_addr":"172.21.15.123:6821/1391966446","state":["exists","up"]},{"osd":8,"uuid":"e918ec1b-7608-4f73-9067-29082ea63999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":2260577139},{"type":"v1","addr":"172.21.15.194:6817","nonce":2260577139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":2260577139},{"type":"v1","addr":"172.21.15.194:6819","nonce":2260577139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":2260577139},{"type":"v1","addr":"172.21.15.194:6823","nonce":2260577139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":2260577139},{"type":"v1","addr":"172.21.15.194:6821","nonce":2260577139}]},"public_addr":"172.21.15.194:6817/2260577139","cluster_addr":"172.21.15.194:6819/2260577139","heartbeat_back_addr":"172.21.15.194:6823/2260577139","heartbeat_front_addr":"172.21.15.194:6821/2260577139","state":["exists","up"]},{"osd":9,"uuid":"0fd76722-3319-4828-ab68-108607a4bd0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":4004843075},{"type":"v1","addr":"172.21.15.19:6827","nonce":4004843075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":4004843075},{"type":"v1","addr":"172.21.15.19:6829","nonce":4004843075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":4004843075},{"type":"v1","addr":"172.21.15.19:6833","nonce":4004843075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":4004843075},{"type":"v1","addr":"172.21.15.19:6831","nonce":4004843075}]},"public_addr":"172.21.15.19:6827/4004843075","cluster_addr":"172.21.15.19:6829/4004843075","heartbeat_back_addr":"172.21.15.19:6833/4004843075","heartbeat_front_addr":"172.21.15.19:6831/4004843075","state":["exists","up"]},{"osd":10,"uuid":"624bb8f5-6d72-43d5-9986-1b48b8a564ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2591845068},{"type":"v1","addr":"172.21.15.123:6825","nonce":2591845068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":2591845068},{"type":"v1","addr":"172.21.15.123:6827","nonce":2591845068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2591845068},{"type":"v1","addr":"172.21.15.123:6831","nonce":2591845068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2591845068},{"type":"v1","addr":"172.21.15.123:6829","nonce":2591845068}]},"public_addr":"172.21.15.123:6825/2591845068","cluster_addr":"172.21.15.123:6827/2591845068","heartbeat_back_addr":"172.21.15.123:6831/2591845068","heartbeat_front_addr":"172.21.15.123:6829/2591845068","state":["exists","up"]},{"osd":11,"uuid":"219a55b3-b01e-471d-90da-1a3043bf3e34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":3985496260},{"type":"v1","addr":"172.21.15.194:6825","nonce":3985496260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":3985496260},{"type":"v1","addr":"172.21.15.194:6827","nonce":3985496260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":3985496260},{"type":"v1","addr":"172.21.15.194:6831","nonce":3985496260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":3985496260},{"type":"v1","addr":"172.21.15.194:6829","nonce":3985496260}]},"public_addr":"172.21.15.194:6825/3985496260","cluster_addr":"172.21.15.194:6827/3985496260","heartbeat_back_addr":"172.21.15.194:6831/3985496260","heartbeat_front_addr":"172.21.15.194:6829/3985496260","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:21:17.278347+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:21:34.469282+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:21:51.139919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:22:09.126458+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:22:26.186801+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:22:43.178997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:23:01.109570+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:23:18.592191+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:23:35.685046+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:23:53.574820+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:24:10.987531+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:24:28.231206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.19:0/1047252857":"2024-01-25T14:20:02.911218+0000","172.21.15.19:6801/3758061162":"2024-01-25T14:20:02.911218+0000","172.21.15.19:6800/3758061162":"2024-01-25T14:20:02.911218+0000","172.21.15.19:6801/2754937713":"2024-01-25T14:19:39.853212+0000","172.21.15.19:0/4062420470":"2024-01-25T14:19:39.853212+0000","172.21.15.19:6800/2754937713":"2024-01-25T14:19:39.853212+0000","172.21.15.19:0/822613692":"2024-01-25T14:20:02.911218+0000","172.21.15.19:0/3970666946":"2024-01-25T14:19:39.853212+0000","172.21.15.19:0/1116053938":"2024-01-25T14:19:39.853212+0000","172.21.15.19:0/1456770388":"2024-01-25T14:20:02.911218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T14:25:17.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.459+0000 7fd601aec700 1 -- 172.21.15.19:0/3908232960 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fd5e40a0570 msgr2=0x7fd5e40a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:17.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.459+0000 7fd601aec700 1 --2- 172.21.15.19:0/3908232960 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fd5e40a0570 0x7fd5e40a2a30 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec011550 tx=0x7fd5ec009260 comp rx=0 tx=0).stop 2024-01-24T14:25:17.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.459+0000 7fd601aec700 1 -- 172.21.15.19:0/3908232960 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5fc104170 msgr2=0x7fd5fc196ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:17.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.459+0000 7fd601aec700 1 --2- 172.21.15.19:0/3908232960 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5fc104170 0x7fd5fc196ee0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd5f0000c00 tx=0x7fd5f0005720 comp rx=0 tx=0).stop 2024-01-24T14:25:17.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.460+0000 7fd601aec700 1 -- 172.21.15.19:0/3908232960 shutdown_connections 2024-01-24T14:25:17.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.460+0000 7fd601aec700 1 --2- 172.21.15.19:0/3908232960 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fd5e40a0570 0x7fd5e40a2a30 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:17.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.460+0000 7fd601aec700 1 --2- 172.21.15.19:0/3908232960 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd5fc101840 0x7fd5fc1969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:17.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.460+0000 7fd601aec700 1 --2- 172.21.15.19:0/3908232960 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd5fc106aa0 0x7fd5fc19df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:17.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.460+0000 7fd601aec700 1 --2- 172.21.15.19:0/3908232960 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5fc104170 0x7fd5fc196ee0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:17.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.460+0000 7fd601aec700 1 -- 172.21.15.19:0/3908232960 >> 172.21.15.19:0/3908232960 conn(0x7fd5fc0fb170 msgr2=0x7fd5fc102cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:17.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.460+0000 7fd601aec700 1 -- 172.21.15.19:0/3908232960 shutdown_connections 2024-01-24T14:25:17.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.460+0000 7fd601aec700 1 -- 172.21.15.19:0/3908232960 wait complete. 2024-01-24T14:25:17.474 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-24T14:21:53.453429+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-24T14:25:01.967538+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-24T14:25:02.864396+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '78', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-01-24T14:25:05.950591+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-01-24T14:25:17.475 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-24T14:25:17.615 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.614+0000 7fa4566c2700 1 -- 172.21.15.19:0/164601033 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa4501017d0 msgr2=0x7fa450108080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:17.616 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.614+0000 7fa4566c2700 1 --2- 172.21.15.19:0/164601033 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa4501017d0 0x7fa450108080 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa44800b0d0 tx=0x7fa44800b3e0 comp rx=0 tx=0).stop 2024-01-24T14:25:17.616 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.614+0000 7fa4566c2700 1 -- 172.21.15.19:0/164601033 shutdown_connections 2024-01-24T14:25:17.616 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.614+0000 7fa4566c2700 1 --2- 172.21.15.19:0/164601033 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa4501017d0 0x7fa450108080 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:17.616 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.614+0000 7fa4566c2700 1 --2- 172.21.15.19:0/164601033 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa4500ffc40 0x7fa450100060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:17.616 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.614+0000 7fa4566c2700 1 --2- 172.21.15.19:0/164601033 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa450100e30 0x7fa450101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:17.616 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.614+0000 7fa4566c2700 1 -- 172.21.15.19:0/164601033 >> 172.21.15.19:0/164601033 conn(0x7fa4500fb1d0 msgr2=0x7fa4500fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:17.616 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.614+0000 7fa4566c2700 1 -- 172.21.15.19:0/164601033 shutdown_connections 2024-01-24T14:25:17.616 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.614+0000 7fa4566c2700 1 -- 172.21.15.19:0/164601033 wait complete. 2024-01-24T14:25:17.616 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.615+0000 7fa4566c2700 1 Processor -- start 2024-01-24T14:25:17.616 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.615+0000 7fa4566c2700 1 -- start start 2024-01-24T14:25:17.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.615+0000 7fa4566c2700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4500ffc40 0x7fa450196ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:17.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.615+0000 7fa4566c2700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa450100e30 0x7fa450196ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:17.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.615+0000 7fa4566c2700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa4501017d0 0x7fa45019e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:17.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.615+0000 7fa4566c2700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45019e630 con 0x7fa450100e30 2024-01-24T14:25:17.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.615+0000 7fa4566c2700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45019e770 con 0x7fa4500ffc40 2024-01-24T14:25:17.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.615+0000 7fa4566c2700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45019e900 con 0x7fa4501017d0 2024-01-24T14:25:17.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.615+0000 7fa44f7fe700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa450100e30 0x7fa450196ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:17.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.615+0000 7fa44ffff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4500ffc40 0x7fa450196ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:17.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.615+0000 7fa44f7fe700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa450100e30 0x7fa450196ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:57772/0 (socket says 172.21.15.19:57772) 2024-01-24T14:25:17.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.615+0000 7fa44f7fe700 1 -- 172.21.15.19:0/2281464420 learned_addr learned my addr 172.21.15.19:0/2281464420 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:25:17.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.615+0000 7fa454c5f700 1 --2- 172.21.15.19:0/2281464420 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa4501017d0 0x7fa45019e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:17.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.616+0000 7fa44f7fe700 1 -- 172.21.15.19:0/2281464420 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa4501017d0 msgr2=0x7fa45019e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:17.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.616+0000 7fa44f7fe700 1 --2- 172.21.15.19:0/2281464420 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa4501017d0 0x7fa45019e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:17.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.616+0000 7fa44f7fe700 1 -- 172.21.15.19:0/2281464420 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4500ffc40 msgr2=0x7fa450196ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:17.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.616+0000 7fa44f7fe700 1 --2- 172.21.15.19:0/2281464420 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4500ffc40 0x7fa450196ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:17.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.616+0000 7fa44f7fe700 1 -- 172.21.15.19:0/2281464420 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa448009d00 con 0x7fa450100e30 2024-01-24T14:25:17.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.616+0000 7fa454c5f700 1 --2- 172.21.15.19:0/2281464420 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa4501017d0 0x7fa45019e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T14:25:17.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.616+0000 7fa44ffff700 1 --2- 172.21.15.19:0/2281464420 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4500ffc40 0x7fa450196ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-24T14:25:17.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.616+0000 7fa44f7fe700 1 --2- 172.21.15.19:0/2281464420 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa450100e30 0x7fa450196ff0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa44400e9a0 tx=0x7fa44400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:17.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.616+0000 7fa44d7fa700 1 -- 172.21.15.19:0/2281464420 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa444009790 con 0x7fa450100e30 2024-01-24T14:25:17.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.616+0000 7fa44d7fa700 1 -- 172.21.15.19:0/2281464420 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa44400cbd0 con 0x7fa450100e30 2024-01-24T14:25:17.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.616+0000 7fa4566c2700 1 -- 172.21.15.19:0/2281464420 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa45019ebe0 con 0x7fa450100e30 2024-01-24T14:25:17.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.616+0000 7fa44d7fa700 1 -- 172.21.15.19:0/2281464420 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa444018680 con 0x7fa450100e30 2024-01-24T14:25:17.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.616+0000 7fa4566c2700 1 -- 172.21.15.19:0/2281464420 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa45019f0f0 con 0x7fa450100e30 2024-01-24T14:25:17.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.617+0000 7fa4566c2700 1 -- 172.21.15.19:0/2281464420 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa45010b4b0 con 0x7fa450100e30 2024-01-24T14:25:17.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.618+0000 7fa44d7fa700 1 -- 172.21.15.19:0/2281464420 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7fa44400cd40 con 0x7fa450100e30 2024-01-24T14:25:17.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.619+0000 7fa44d7fa700 1 --2- 172.21.15.19:0/2281464420 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fa4380a0520 0x7fa4380a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:17.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.619+0000 7fa44ffff700 1 --2- 172.21.15.19:0/2281464420 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fa4380a0520 0x7fa4380a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:17.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.619+0000 7fa44d7fa700 1 -- 172.21.15.19:0/2281464420 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fa444014070 con 0x7fa450100e30 2024-01-24T14:25:17.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.620+0000 7fa44ffff700 1 --2- 172.21.15.19:0/2281464420 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fa4380a0520 0x7fa4380a29e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa450100c90 tx=0x7fa44000a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:17.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.624+0000 7fa44d7fa700 1 -- 172.21.15.19:0/2281464420 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fa444052120 con 0x7fa450100e30 2024-01-24T14:25:17.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:17 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/1477010470' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-01-24T14:25:17.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:17 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/2127339402' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-01-24T14:25:17.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:17 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/1477010470' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-01-24T14:25:17.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:17 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/2127339402' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-01-24T14:25:17.834 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.833+0000 7fa4566c2700 1 -- 172.21.15.19:0/2281464420 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa45019f6a0 con 0x7fa450100e30 2024-01-24T14:25:17.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.834+0000 7fa44d7fa700 1 -- 172.21.15.19:0/2281464420 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa4440957a0 con 0x7fa450100e30 2024-01-24T14:25:17.835 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 1 2024-01-24T14:25:17.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.836+0000 7fa4566c2700 1 -- 172.21.15.19:0/2281464420 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fa4380a0520 msgr2=0x7fa4380a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:17.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.836+0000 7fa4566c2700 1 --2- 172.21.15.19:0/2281464420 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fa4380a0520 0x7fa4380a29e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa450100c90 tx=0x7fa44000a3b0 comp rx=0 tx=0).stop 2024-01-24T14:25:17.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.836+0000 7fa4566c2700 1 -- 172.21.15.19:0/2281464420 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa450100e30 msgr2=0x7fa450196ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:17.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.836+0000 7fa4566c2700 1 --2- 172.21.15.19:0/2281464420 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa450100e30 0x7fa450196ff0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa44400e9a0 tx=0x7fa44400ecb0 comp rx=0 tx=0).stop 2024-01-24T14:25:17.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.836+0000 7fa4566c2700 1 -- 172.21.15.19:0/2281464420 shutdown_connections 2024-01-24T14:25:17.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.836+0000 7fa4566c2700 1 --2- 172.21.15.19:0/2281464420 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa4501017d0 0x7fa45019e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:17.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.836+0000 7fa4566c2700 1 --2- 172.21.15.19:0/2281464420 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa4500ffc40 0x7fa450196ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:17.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.836+0000 7fa4566c2700 1 --2- 172.21.15.19:0/2281464420 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fa4380a0520 0x7fa4380a29e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:17.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.836+0000 7fa4566c2700 1 --2- 172.21.15.19:0/2281464420 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa450100e30 0x7fa450196ff0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:17.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.836+0000 7fa4566c2700 1 -- 172.21.15.19:0/2281464420 >> 172.21.15.19:0/2281464420 conn(0x7fa4500fb1d0 msgr2=0x7fa450068ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:17.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.836+0000 7fa4566c2700 1 -- 172.21.15.19:0/2281464420 shutdown_connections 2024-01-24T14:25:17.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.836+0000 7fa4566c2700 1 -- 172.21.15.19:0/2281464420 wait complete. 2024-01-24T14:25:17.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:17 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/1477010470' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-01-24T14:25:17.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:17 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/2127339402' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-01-24T14:25:17.851 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-24T14:25:17.991 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.990+0000 7f8e30bbb700 1 -- 172.21.15.19:0/2472227124 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8e2c1040e0 msgr2=0x7f8e2c1064d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:17.991 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.990+0000 7f8e30bbb700 1 --2- 172.21.15.19:0/2472227124 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8e2c1040e0 0x7f8e2c1064d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8e20009990 tx=0x7f8e20009ca0 comp rx=0 tx=0).stop 2024-01-24T14:25:17.992 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.990+0000 7f8e30bbb700 1 -- 172.21.15.19:0/2472227124 shutdown_connections 2024-01-24T14:25:17.992 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.990+0000 7f8e30bbb700 1 --2- 172.21.15.19:0/2472227124 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e2c106a10 0x7f8e2c108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:17.992 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.990+0000 7f8e30bbb700 1 --2- 172.21.15.19:0/2472227124 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8e2c1017b0 0x7f8e2c103ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:17.992 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.990+0000 7f8e30bbb700 1 --2- 172.21.15.19:0/2472227124 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8e2c1040e0 0x7f8e2c1064d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:17.992 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.990+0000 7f8e30bbb700 1 -- 172.21.15.19:0/2472227124 >> 172.21.15.19:0/2472227124 conn(0x7f8e2c0fb1a0 msgr2=0x7f8e2c0fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:17.992 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.990+0000 7f8e30bbb700 1 -- 172.21.15.19:0/2472227124 shutdown_connections 2024-01-24T14:25:17.992 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.990+0000 7f8e30bbb700 1 -- 172.21.15.19:0/2472227124 wait complete. 2024-01-24T14:25:17.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.991+0000 7f8e30bbb700 1 Processor -- start 2024-01-24T14:25:17.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.991+0000 7f8e30bbb700 1 -- start start 2024-01-24T14:25:17.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.991+0000 7f8e30bbb700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8e2c1017b0 0x7f8e2c196a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:17.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.991+0000 7f8e30bbb700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8e2c1040e0 0x7f8e2c196f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:17.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.991+0000 7f8e30bbb700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e2c106a10 0x7f8e2c19dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:17.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.991+0000 7f8e30bbb700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2c19e5a0 con 0x7f8e2c1017b0 2024-01-24T14:25:17.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.991+0000 7f8e30bbb700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2c19e6e0 con 0x7f8e2c106a10 2024-01-24T14:25:17.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.991+0000 7f8e30bbb700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2c19e870 con 0x7f8e2c1040e0 2024-01-24T14:25:17.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.991+0000 7f8e2a59c700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8e2c1017b0 0x7f8e2c196a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:17.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.991+0000 7f8e2a59c700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8e2c1017b0 0x7f8e2c196a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:57796/0 (socket says 172.21.15.19:57796) 2024-01-24T14:25:17.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.991+0000 7f8e2a59c700 1 -- 172.21.15.19:0/815059503 learned_addr learned my addr 172.21.15.19:0/815059503 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:25:17.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.991+0000 7f8e2ad9d700 1 --2- 172.21.15.19:0/815059503 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e2c106a10 0x7f8e2c19dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:17.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.991+0000 7f8e29d9b700 1 --2- 172.21.15.19:0/815059503 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8e2c1040e0 0x7f8e2c196f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:17.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.992+0000 7f8e2a59c700 1 -- 172.21.15.19:0/815059503 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8e2c1040e0 msgr2=0x7f8e2c196f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:17.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.992+0000 7f8e2a59c700 1 --2- 172.21.15.19:0/815059503 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8e2c1040e0 0x7f8e2c196f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:17.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.992+0000 7f8e2a59c700 1 -- 172.21.15.19:0/815059503 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e2c106a10 msgr2=0x7f8e2c19dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:17.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.992+0000 7f8e2a59c700 1 --2- 172.21.15.19:0/815059503 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e2c106a10 0x7f8e2c19dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:17.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.992+0000 7f8e2a59c700 1 -- 172.21.15.19:0/815059503 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e200095f0 con 0x7f8e2c1017b0 2024-01-24T14:25:17.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.992+0000 7f8e2a59c700 1 --2- 172.21.15.19:0/815059503 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8e2c1017b0 0x7f8e2c196a20 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c00e9a0 tx=0x7f8e1c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:17.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.992+0000 7f8e1b7fe700 1 -- 172.21.15.19:0/815059503 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e1c009790 con 0x7f8e2c1017b0 2024-01-24T14:25:17.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.992+0000 7f8e30bbb700 1 -- 172.21.15.19:0/815059503 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e2c19eb50 con 0x7f8e2c1017b0 2024-01-24T14:25:17.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.992+0000 7f8e1b7fe700 1 -- 172.21.15.19:0/815059503 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8e1c010430 con 0x7f8e2c1017b0 2024-01-24T14:25:17.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.992+0000 7f8e1b7fe700 1 -- 172.21.15.19:0/815059503 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e1c018680 con 0x7f8e2c1017b0 2024-01-24T14:25:17.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.992+0000 7f8e30bbb700 1 -- 172.21.15.19:0/815059503 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e2c19f060 con 0x7f8e2c1017b0 2024-01-24T14:25:17.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.993+0000 7f8e30bbb700 1 -- 172.21.15.19:0/815059503 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e2c0fcd20 con 0x7f8e2c1017b0 2024-01-24T14:25:17.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.994+0000 7f8e1b7fe700 1 -- 172.21.15.19:0/815059503 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f8e1c0105a0 con 0x7f8e2c1017b0 2024-01-24T14:25:17.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.995+0000 7f8e1b7fe700 1 --2- 172.21.15.19:0/815059503 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f8e140a0520 0x7f8e140a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:17.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.995+0000 7f8e1b7fe700 1 -- 172.21.15.19:0/815059503 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f8e1c014070 con 0x7f8e2c1017b0 2024-01-24T14:25:17.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.995+0000 7f8e29d9b700 1 --2- 172.21.15.19:0/815059503 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f8e140a0520 0x7f8e140a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:17.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.995+0000 7f8e29d9b700 1 --2- 172.21.15.19:0/815059503 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f8e140a0520 0x7f8e140a29e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8e2000b2e0 tx=0x7f8e20005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:17.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:17.998+0000 7f8e1b7fe700 1 -- 172.21.15.19:0/815059503 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f8e1c0958b0 con 0x7f8e2c1017b0 2024-01-24T14:25:18.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.209+0000 7f8e30bbb700 1 -- 172.21.15.19:0/815059503 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8e2c02cf20 con 0x7f8e2c1017b0 2024-01-24T14:25:18.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.209+0000 7f8e1b7fe700 1 -- 172.21.15.19:0/815059503 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v82) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8e1c098ed0 con 0x7f8e2c1017b0 2024-01-24T14:25:18.210 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 32 2024-01-24T14:25:18.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.212+0000 7f8e30bbb700 1 -- 172.21.15.19:0/815059503 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f8e140a0520 msgr2=0x7f8e140a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:18.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.212+0000 7f8e30bbb700 1 --2- 172.21.15.19:0/815059503 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f8e140a0520 0x7f8e140a29e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8e2000b2e0 tx=0x7f8e20005fb0 comp rx=0 tx=0).stop 2024-01-24T14:25:18.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.212+0000 7f8e30bbb700 1 -- 172.21.15.19:0/815059503 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8e2c1017b0 msgr2=0x7f8e2c196a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:18.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.212+0000 7f8e30bbb700 1 --2- 172.21.15.19:0/815059503 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8e2c1017b0 0x7f8e2c196a20 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c00e9a0 tx=0x7f8e1c00ecb0 comp rx=0 tx=0).stop 2024-01-24T14:25:18.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.212+0000 7f8e30bbb700 1 -- 172.21.15.19:0/815059503 shutdown_connections 2024-01-24T14:25:18.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.212+0000 7f8e30bbb700 1 --2- 172.21.15.19:0/815059503 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f8e140a0520 0x7f8e140a29e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:18.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.212+0000 7f8e30bbb700 1 --2- 172.21.15.19:0/815059503 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8e2c1017b0 0x7f8e2c196a20 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:18.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.212+0000 7f8e30bbb700 1 --2- 172.21.15.19:0/815059503 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e2c106a10 0x7f8e2c19dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:18.214 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.212+0000 7f8e30bbb700 1 --2- 172.21.15.19:0/815059503 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8e2c1040e0 0x7f8e2c196f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:18.214 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.212+0000 7f8e30bbb700 1 -- 172.21.15.19:0/815059503 >> 172.21.15.19:0/815059503 conn(0x7f8e2c0fb1a0 msgr2=0x7f8e2c102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:18.214 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.212+0000 7f8e30bbb700 1 -- 172.21.15.19:0/815059503 shutdown_connections 2024-01-24T14:25:18.214 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.212+0000 7f8e30bbb700 1 -- 172.21.15.19:0/815059503 wait complete. 2024-01-24T14:25:18.226 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-24T14:25:18.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.366+0000 7fde7d915700 1 -- 172.21.15.19:0/3459615183 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fde78102a60 msgr2=0x7fde78109310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:18.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.366+0000 7fde7d915700 1 --2- 172.21.15.19:0/3459615183 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fde78102a60 0x7fde78109310 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fde7000b0a0 tx=0x7fde7000b3b0 comp rx=0 tx=0).stop 2024-01-24T14:25:18.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.366+0000 7fde7d915700 1 -- 172.21.15.19:0/3459615183 shutdown_connections 2024-01-24T14:25:18.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.366+0000 7fde7d915700 1 --2- 172.21.15.19:0/3459615183 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fde78102a60 0x7fde78109310 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:18.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.366+0000 7fde7d915700 1 --2- 172.21.15.19:0/3459615183 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fde780ff050 0x7fde780ff470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:18.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.366+0000 7fde7d915700 1 --2- 172.21.15.19:0/3459615183 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde781020c0 0x7fde78102520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:18.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.366+0000 7fde7d915700 1 -- 172.21.15.19:0/3459615183 >> 172.21.15.19:0/3459615183 conn(0x7fde780fa960 msgr2=0x7fde780fcda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:18.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.366+0000 7fde7d915700 1 -- 172.21.15.19:0/3459615183 shutdown_connections 2024-01-24T14:25:18.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.366+0000 7fde7d915700 1 -- 172.21.15.19:0/3459615183 wait complete. 2024-01-24T14:25:18.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.366+0000 7fde7d915700 1 Processor -- start 2024-01-24T14:25:18.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.367+0000 7fde7d915700 1 -- start start 2024-01-24T14:25:18.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.367+0000 7fde7d915700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fde780ff050 0x7fde78196b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:18.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.367+0000 7fde7d915700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fde781020c0 0x7fde78197080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:18.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.367+0000 7fde7d915700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde78102a60 0x7fde7819e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:18.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.367+0000 7fde7d915700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7819e6c0 con 0x7fde780ff050 2024-01-24T14:25:18.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.367+0000 7fde7d915700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7819e800 con 0x7fde78102a60 2024-01-24T14:25:18.370 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.367+0000 7fde7d915700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7819e990 con 0x7fde781020c0 2024-01-24T14:25:18.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.367+0000 7fde76ffd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fde780ff050 0x7fde78196b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:18.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.367+0000 7fde76ffd700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fde780ff050 0x7fde78196b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:57820/0 (socket says 172.21.15.19:57820) 2024-01-24T14:25:18.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.367+0000 7fde76ffd700 1 -- 172.21.15.19:0/3720100070 learned_addr learned my addr 172.21.15.19:0/3720100070 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:25:18.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.367+0000 7fde777fe700 1 --2- 172.21.15.19:0/3720100070 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde78102a60 0x7fde7819e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:18.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.367+0000 7fde767fc700 1 --2- 172.21.15.19:0/3720100070 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fde781020c0 0x7fde78197080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:18.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.367+0000 7fde76ffd700 1 -- 172.21.15.19:0/3720100070 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fde781020c0 msgr2=0x7fde78197080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:18.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.367+0000 7fde76ffd700 1 --2- 172.21.15.19:0/3720100070 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fde781020c0 0x7fde78197080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:18.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.367+0000 7fde76ffd700 1 -- 172.21.15.19:0/3720100070 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde78102a60 msgr2=0x7fde7819e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:18.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.367+0000 7fde76ffd700 1 --2- 172.21.15.19:0/3720100070 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde78102a60 0x7fde7819e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:18.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.367+0000 7fde76ffd700 1 -- 172.21.15.19:0/3720100070 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde70009d00 con 0x7fde780ff050 2024-01-24T14:25:18.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.368+0000 7fde76ffd700 1 --2- 172.21.15.19:0/3720100070 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fde780ff050 0x7fde78196b40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fde6800d980 tx=0x7fde6800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:18.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.368+0000 7fde7c913700 1 -- 172.21.15.19:0/3720100070 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde68004020 con 0x7fde780ff050 2024-01-24T14:25:18.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.368+0000 7fde7c913700 1 -- 172.21.15.19:0/3720100070 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fde68009730 con 0x7fde780ff050 2024-01-24T14:25:18.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.368+0000 7fde7c913700 1 -- 172.21.15.19:0/3720100070 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde68003a80 con 0x7fde780ff050 2024-01-24T14:25:18.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.368+0000 7fde7d915700 1 -- 172.21.15.19:0/3720100070 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde7819ec70 con 0x7fde780ff050 2024-01-24T14:25:18.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.368+0000 7fde7d915700 1 -- 172.21.15.19:0/3720100070 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde7819f130 con 0x7fde780ff050 2024-01-24T14:25:18.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.369+0000 7fde7d915700 1 -- 172.21.15.19:0/3720100070 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde7810c5c0 con 0x7fde780ff050 2024-01-24T14:25:18.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.370+0000 7fde7c913700 1 -- 172.21.15.19:0/3720100070 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7fde680098a0 con 0x7fde780ff050 2024-01-24T14:25:18.373 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.371+0000 7fde7c913700 1 --2- 172.21.15.19:0/3720100070 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fde600a0520 0x7fde600a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:18.373 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.371+0000 7fde767fc700 1 --2- 172.21.15.19:0/3720100070 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fde600a0520 0x7fde600a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:18.373 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.371+0000 7fde7c913700 1 -- 172.21.15.19:0/3720100070 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fde680cbc10 con 0x7fde780ff050 2024-01-24T14:25:18.373 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.371+0000 7fde767fc700 1 --2- 172.21.15.19:0/3720100070 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fde600a0520 0x7fde600a29e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fde6c005950 tx=0x7fde6c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:18.375 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.374+0000 7fde7c913700 1 -- 172.21.15.19:0/3720100070 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7fde68095560 con 0x7fde780ff050 2024-01-24T14:25:18.485 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:18 smithi019 ceph-mon[40663]: pgmap v229: 129 pgs: 129 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 864 B/s wr, 3 op/s 2024-01-24T14:25:18.485 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:18 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/3908232960' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T14:25:18.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:18 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/2281464420' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-01-24T14:25:18.486 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:18 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/815059503' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-01-24T14:25:18.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.587+0000 7fde7d915700 1 -- 172.21.15.19:0/3720100070 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fde78107d80 con 0x7fde780ff050 2024-01-24T14:25:18.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.588+0000 7fde7c913700 1 -- 172.21.15.19:0/3720100070 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v82) v1 ==== 100+0+11 (secure 0 0 0) 0x7fde68098b80 con 0x7fde780ff050 2024-01-24T14:25:18.589 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 64 2024-01-24T14:25:18.591 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.590+0000 7fde7d915700 1 -- 172.21.15.19:0/3720100070 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fde600a0520 msgr2=0x7fde600a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:18.592 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.590+0000 7fde7d915700 1 --2- 172.21.15.19:0/3720100070 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fde600a0520 0x7fde600a29e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fde6c005950 tx=0x7fde6c0058e0 comp rx=0 tx=0).stop 2024-01-24T14:25:18.592 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.590+0000 7fde7d915700 1 -- 172.21.15.19:0/3720100070 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fde780ff050 msgr2=0x7fde78196b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:18.592 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.590+0000 7fde7d915700 1 --2- 172.21.15.19:0/3720100070 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fde780ff050 0x7fde78196b40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fde6800d980 tx=0x7fde6800dc90 comp rx=0 tx=0).stop 2024-01-24T14:25:18.592 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.590+0000 7fde7d915700 1 -- 172.21.15.19:0/3720100070 shutdown_connections 2024-01-24T14:25:18.592 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.590+0000 7fde7d915700 1 --2- 172.21.15.19:0/3720100070 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7fde600a0520 0x7fde600a29e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:18.592 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.590+0000 7fde7d915700 1 --2- 172.21.15.19:0/3720100070 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fde780ff050 0x7fde78196b40 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:18.592 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.590+0000 7fde7d915700 1 --2- 172.21.15.19:0/3720100070 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde78102a60 0x7fde7819e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:18.592 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.590+0000 7fde7d915700 1 --2- 172.21.15.19:0/3720100070 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fde781020c0 0x7fde78197080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:18.592 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.590+0000 7fde7d915700 1 -- 172.21.15.19:0/3720100070 >> 172.21.15.19:0/3720100070 conn(0x7fde780fa960 msgr2=0x7fde780fcb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:18.592 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.590+0000 7fde7d915700 1 -- 172.21.15.19:0/3720100070 shutdown_connections 2024-01-24T14:25:18.592 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.590+0000 7fde7d915700 1 -- 172.21.15.19:0/3720100070 wait complete. 2024-01-24T14:25:18.605 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-01-24T14:25:18.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.745+0000 7f1262031700 1 -- 172.21.15.19:0/3178359223 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f125c104170 msgr2=0x7f125c106560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:18.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.745+0000 7f1262031700 1 --2- 172.21.15.19:0/3178359223 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f125c104170 0x7f125c106560 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1250009990 tx=0x7f1250009ca0 comp rx=0 tx=0).stop 2024-01-24T14:25:18.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.745+0000 7f1262031700 1 -- 172.21.15.19:0/3178359223 shutdown_connections 2024-01-24T14:25:18.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.745+0000 7f1262031700 1 --2- 172.21.15.19:0/3178359223 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f125c106aa0 0x7f125c108eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:18.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.745+0000 7f1262031700 1 --2- 172.21.15.19:0/3178359223 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f125c101840 0x7f125c103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:18.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.745+0000 7f1262031700 1 --2- 172.21.15.19:0/3178359223 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f125c104170 0x7f125c106560 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:18.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.745+0000 7f1262031700 1 -- 172.21.15.19:0/3178359223 >> 172.21.15.19:0/3178359223 conn(0x7f125c0fb1d0 msgr2=0x7f125c0fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:18.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.745+0000 7f1262031700 1 -- 172.21.15.19:0/3178359223 shutdown_connections 2024-01-24T14:25:18.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.745+0000 7f1262031700 1 -- 172.21.15.19:0/3178359223 wait complete. 2024-01-24T14:25:18.749 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.746+0000 7f1262031700 1 Processor -- start 2024-01-24T14:25:18.749 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.746+0000 7f1262031700 1 -- start start 2024-01-24T14:25:18.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.746+0000 7f1262031700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f125c101840 0x7f125c1969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:18.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.746+0000 7f1262031700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f125c104170 0x7f125c196f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:18.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.746+0000 7f1262031700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f125c106aa0 0x7f125c19dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:18.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.746+0000 7f1262031700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f125c19e570 con 0x7f125c101840 2024-01-24T14:25:18.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.746+0000 7f1262031700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f125c19e6b0 con 0x7f125c104170 2024-01-24T14:25:18.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.746+0000 7f1262031700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f125c19e840 con 0x7f125c106aa0 2024-01-24T14:25:18.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.747+0000 7f125b7fe700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f125c101840 0x7f125c1969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:18.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.747+0000 7f125b7fe700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f125c101840 0x7f125c1969f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:57852/0 (socket says 172.21.15.19:57852) 2024-01-24T14:25:18.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.747+0000 7f125b7fe700 1 -- 172.21.15.19:0/3177533848 learned_addr learned my addr 172.21.15.19:0/3177533848 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:25:18.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.747+0000 7f125affd700 1 --2- 172.21.15.19:0/3177533848 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f125c104170 0x7f125c196f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:18.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.747+0000 7f125bfff700 1 --2- 172.21.15.19:0/3177533848 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f125c106aa0 0x7f125c19dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:18.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.747+0000 7f125b7fe700 1 -- 172.21.15.19:0/3177533848 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f125c106aa0 msgr2=0x7f125c19dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:18.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.747+0000 7f125b7fe700 1 --2- 172.21.15.19:0/3177533848 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f125c106aa0 0x7f125c19dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:18.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.747+0000 7f125b7fe700 1 -- 172.21.15.19:0/3177533848 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f125c104170 msgr2=0x7f125c196f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:18.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.747+0000 7f125b7fe700 1 --2- 172.21.15.19:0/3177533848 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f125c104170 0x7f125c196f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:18.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.747+0000 7f125b7fe700 1 -- 172.21.15.19:0/3177533848 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12500095f0 con 0x7f125c101840 2024-01-24T14:25:18.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.747+0000 7f125affd700 1 --2- 172.21.15.19:0/3177533848 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f125c104170 0x7f125c196f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T14:25:18.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.747+0000 7f125b7fe700 1 --2- 172.21.15.19:0/3177533848 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f125c101840 0x7f125c1969f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f124c00d980 tx=0x7f124c00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:18.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.747+0000 7f1258ff9700 1 -- 172.21.15.19:0/3177533848 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f124c00bb60 con 0x7f125c101840 2024-01-24T14:25:18.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.747+0000 7f1258ff9700 1 -- 172.21.15.19:0/3177533848 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f124c007640 con 0x7f125c101840 2024-01-24T14:25:18.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.747+0000 7f1262031700 1 -- 172.21.15.19:0/3177533848 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f125c19eb20 con 0x7f125c101840 2024-01-24T14:25:18.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.747+0000 7f1258ff9700 1 -- 172.21.15.19:0/3177533848 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f124c004e90 con 0x7f125c101840 2024-01-24T14:25:18.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.747+0000 7f1262031700 1 -- 172.21.15.19:0/3177533848 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f125c19f030 con 0x7f125c101840 2024-01-24T14:25:18.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.749+0000 7f1262031700 1 -- 172.21.15.19:0/3177533848 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f125c10c550 con 0x7f125c101840 2024-01-24T14:25:18.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.750+0000 7f1258ff9700 1 -- 172.21.15.19:0/3177533848 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f124c00bcc0 con 0x7f125c101840 2024-01-24T14:25:18.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.750+0000 7f1258ff9700 1 --2- 172.21.15.19:0/3177533848 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f12440a0520 0x7f12440a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:18.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.751+0000 7f1258ff9700 1 -- 172.21.15.19:0/3177533848 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f124c0cbff0 con 0x7f125c101840 2024-01-24T14:25:18.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.751+0000 7f125affd700 1 --2- 172.21.15.19:0/3177533848 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f12440a0520 0x7f12440a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:18.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.751+0000 7f125affd700 1 --2- 172.21.15.19:0/3177533848 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f12440a0520 0x7f12440a29e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1250006010 tx=0x7f125000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:18.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.754+0000 7f1258ff9700 1 -- 172.21.15.19:0/3177533848 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f124c095940 con 0x7f125c101840 2024-01-24T14:25:18.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:18 smithi123 ceph-mon[44966]: pgmap v229: 129 pgs: 129 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 864 B/s wr, 3 op/s 2024-01-24T14:25:18.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:18 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/3908232960' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T14:25:18.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:18 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/2281464420' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-01-24T14:25:18.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:18 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/815059503' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-01-24T14:25:18.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:18 smithi194 ceph-mon[45450]: pgmap v229: 129 pgs: 129 active+clean; 579 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 864 B/s wr, 3 op/s 2024-01-24T14:25:18.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:18 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/3908232960' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T14:25:18.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:18 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/2281464420' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-01-24T14:25:18.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:18 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/815059503' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-01-24T14:25:18.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.967+0000 7f1262031700 1 -- 172.21.15.19:0/3177533848 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f125c19f5e0 con 0x7f125c101840 2024-01-24T14:25:18.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.967+0000 7f1258ff9700 1 -- 172.21.15.19:0/3177533848 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v82) v1 ==== 103+0+11 (secure 0 0 0) 0x7f124c0160a0 con 0x7f125c101840 2024-01-24T14:25:18.968 INFO:teuthology.orchestra.run.smithi019.stdout:pg_num: 32 2024-01-24T14:25:18.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.969+0000 7f1262031700 1 -- 172.21.15.19:0/3177533848 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f12440a0520 msgr2=0x7f12440a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:18.971 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.969+0000 7f1262031700 1 --2- 172.21.15.19:0/3177533848 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f12440a0520 0x7f12440a29e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1250006010 tx=0x7f125000b540 comp rx=0 tx=0).stop 2024-01-24T14:25:18.971 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.969+0000 7f1262031700 1 -- 172.21.15.19:0/3177533848 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f125c101840 msgr2=0x7f125c1969f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:18.971 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.969+0000 7f1262031700 1 --2- 172.21.15.19:0/3177533848 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f125c101840 0x7f125c1969f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f124c00d980 tx=0x7f124c00dd40 comp rx=0 tx=0).stop 2024-01-24T14:25:18.971 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.969+0000 7f1262031700 1 -- 172.21.15.19:0/3177533848 shutdown_connections 2024-01-24T14:25:18.971 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.969+0000 7f1262031700 1 --2- 172.21.15.19:0/3177533848 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f12440a0520 0x7f12440a29e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:18.971 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.969+0000 7f1262031700 1 --2- 172.21.15.19:0/3177533848 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f125c101840 0x7f125c1969f0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:18.971 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.969+0000 7f1262031700 1 --2- 172.21.15.19:0/3177533848 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f125c106aa0 0x7f125c19dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:18.971 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.969+0000 7f1262031700 1 --2- 172.21.15.19:0/3177533848 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f125c104170 0x7f125c196f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:18.971 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.969+0000 7f1262031700 1 -- 172.21.15.19:0/3177533848 >> 172.21.15.19:0/3177533848 conn(0x7f125c0fb1d0 msgr2=0x7f125c102cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:18.971 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.970+0000 7f1262031700 1 -- 172.21.15.19:0/3177533848 shutdown_connections 2024-01-24T14:25:18.971 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:18.970+0000 7f1262031700 1 -- 172.21.15.19:0/3177533848 wait complete. 2024-01-24T14:25:18.984 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-24T14:25:19.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.122+0000 7f5642e5e700 1 -- 172.21.15.19:0/1047116148 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f563c101f50 msgr2=0x7f563c1023b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:19.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.122+0000 7f5642e5e700 1 --2- 172.21.15.19:0/1047116148 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f563c101f50 0x7f563c1023b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f56280099c0 tx=0x7f5628009cd0 comp rx=0 tx=0).stop 2024-01-24T14:25:19.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.122+0000 7f5642e5e700 1 -- 172.21.15.19:0/1047116148 shutdown_connections 2024-01-24T14:25:19.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.122+0000 7f5642e5e700 1 --2- 172.21.15.19:0/1047116148 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f563c1028f0 0x7f563c1091a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:19.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.122+0000 7f5642e5e700 1 --2- 172.21.15.19:0/1047116148 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f563c100d60 0x7f563c101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:19.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.122+0000 7f5642e5e700 1 --2- 172.21.15.19:0/1047116148 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f563c101f50 0x7f563c1023b0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:19.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.122+0000 7f5642e5e700 1 -- 172.21.15.19:0/1047116148 >> 172.21.15.19:0/1047116148 conn(0x7f563c0fc2b0 msgr2=0x7f563c0fe710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:19.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.123+0000 7f5642e5e700 1 -- 172.21.15.19:0/1047116148 shutdown_connections 2024-01-24T14:25:19.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.123+0000 7f5642e5e700 1 -- 172.21.15.19:0/1047116148 wait complete. 2024-01-24T14:25:19.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.123+0000 7f5642e5e700 1 Processor -- start 2024-01-24T14:25:19.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.123+0000 7f5642e5e700 1 -- start start 2024-01-24T14:25:19.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.123+0000 7f5642e5e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f563c100d60 0x7f563c196a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:19.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.123+0000 7f5642e5e700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f563c101f50 0x7f563c196f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:19.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.123+0000 7f5642e5e700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f563c1028f0 0x7f563c19e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:19.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.123+0000 7f5642e5e700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f563c19e5d0 con 0x7f563c101f50 2024-01-24T14:25:19.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.123+0000 7f5642e5e700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f563c19e710 con 0x7f563c1028f0 2024-01-24T14:25:19.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.123+0000 7f5642e5e700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f563c19e8a0 con 0x7f563c100d60 2024-01-24T14:25:19.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.124+0000 7f563bfff700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f563c101f50 0x7f563c196f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:19.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.124+0000 7f563bfff700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f563c101f50 0x7f563c196f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:57876/0 (socket says 172.21.15.19:57876) 2024-01-24T14:25:19.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.124+0000 7f563bfff700 1 -- 172.21.15.19:0/2401630042 learned_addr learned my addr 172.21.15.19:0/2401630042 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:25:19.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.124+0000 7f56413fb700 1 --2- 172.21.15.19:0/2401630042 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f563c1028f0 0x7f563c19e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:19.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.124+0000 7f5640bfa700 1 --2- 172.21.15.19:0/2401630042 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f563c100d60 0x7f563c196a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:19.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.124+0000 7f563bfff700 1 -- 172.21.15.19:0/2401630042 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f563c100d60 msgr2=0x7f563c196a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:19.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.124+0000 7f563bfff700 1 --2- 172.21.15.19:0/2401630042 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f563c100d60 0x7f563c196a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:19.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.124+0000 7f563bfff700 1 -- 172.21.15.19:0/2401630042 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f563c1028f0 msgr2=0x7f563c19e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:19.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.124+0000 7f563bfff700 1 --2- 172.21.15.19:0/2401630042 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f563c1028f0 0x7f563c19e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:19.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.124+0000 7f563bfff700 1 -- 172.21.15.19:0/2401630042 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56280095f0 con 0x7f563c101f50 2024-01-24T14:25:19.128 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.124+0000 7f563bfff700 1 --2- 172.21.15.19:0/2401630042 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f563c101f50 0x7f563c196f90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5628009990 tx=0x7f562800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:19.128 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.124+0000 7f5639ffb700 1 -- 172.21.15.19:0/2401630042 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f562801d070 con 0x7f563c101f50 2024-01-24T14:25:19.128 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.124+0000 7f5639ffb700 1 -- 172.21.15.19:0/2401630042 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f56280054f0 con 0x7f563c101f50 2024-01-24T14:25:19.128 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.124+0000 7f5642e5e700 1 -- 172.21.15.19:0/2401630042 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f563c19eb20 con 0x7f563c101f50 2024-01-24T14:25:19.128 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.124+0000 7f5639ffb700 1 -- 172.21.15.19:0/2401630042 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5628004bd0 con 0x7f563c101f50 2024-01-24T14:25:19.128 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.125+0000 7f5642e5e700 1 -- 172.21.15.19:0/2401630042 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f563c19efa0 con 0x7f563c101f50 2024-01-24T14:25:19.129 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.126+0000 7f5642e5e700 1 -- 172.21.15.19:0/2401630042 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f563c108300 con 0x7f563c101f50 2024-01-24T14:25:19.129 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.127+0000 7f5639ffb700 1 -- 172.21.15.19:0/2401630042 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f562800fb80 con 0x7f563c101f50 2024-01-24T14:25:19.129 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.127+0000 7f5639ffb700 1 --2- 172.21.15.19:0/2401630042 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f562c0a05a0 0x7f562c0a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:19.129 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.128+0000 7f5640bfa700 1 --2- 172.21.15.19:0/2401630042 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f562c0a05a0 0x7f562c0a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:19.129 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.128+0000 7f5639ffb700 1 -- 172.21.15.19:0/2401630042 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f56280ccdc0 con 0x7f563c101f50 2024-01-24T14:25:19.129 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.128+0000 7f5640bfa700 1 --2- 172.21.15.19:0/2401630042 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f562c0a05a0 0x7f562c0a2a60 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5630005950 tx=0x7f56300058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:19.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.131+0000 7f5639ffb700 1 -- 172.21.15.19:0/2401630042 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f56280967a0 con 0x7f563c101f50 2024-01-24T14:25:19.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.362+0000 7f5642e5e700 1 -- 172.21.15.19:0/2401630042 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f563c02cc70 con 0x7f563c101f50 2024-01-24T14:25:19.364 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.363+0000 7f5639ffb700 1 -- 172.21.15.19:0/2401630042 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v82) v1 ==== 81+0+591 (secure 0 0 0) 0x7f5628099dc0 con 0x7f563c101f50 2024-01-24T14:25:19.364 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:0/1047252857 2024-01-25T14:20:02.911218+0000 2024-01-24T14:25:19.364 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:6801/3758061162 2024-01-25T14:20:02.911218+0000 2024-01-24T14:25:19.364 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:6800/3758061162 2024-01-25T14:20:02.911218+0000 2024-01-24T14:25:19.364 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:6801/2754937713 2024-01-25T14:19:39.853212+0000 2024-01-24T14:25:19.364 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:0/4062420470 2024-01-25T14:19:39.853212+0000 2024-01-24T14:25:19.364 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:6800/2754937713 2024-01-25T14:19:39.853212+0000 2024-01-24T14:25:19.364 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:0/822613692 2024-01-25T14:20:02.911218+0000 2024-01-24T14:25:19.365 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:0/3970666946 2024-01-25T14:19:39.853212+0000 2024-01-24T14:25:19.365 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:0/1116053938 2024-01-25T14:19:39.853212+0000 2024-01-24T14:25:19.365 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19:0/1456770388 2024-01-25T14:20:02.911218+0000 2024-01-24T14:25:19.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.365+0000 7f5642e5e700 1 -- 172.21.15.19:0/2401630042 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f562c0a05a0 msgr2=0x7f562c0a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:19.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.365+0000 7f5642e5e700 1 --2- 172.21.15.19:0/2401630042 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f562c0a05a0 0x7f562c0a2a60 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5630005950 tx=0x7f56300058e0 comp rx=0 tx=0).stop 2024-01-24T14:25:19.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.365+0000 7f5642e5e700 1 -- 172.21.15.19:0/2401630042 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f563c101f50 msgr2=0x7f563c196f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:19.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.365+0000 7f5642e5e700 1 --2- 172.21.15.19:0/2401630042 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f563c101f50 0x7f563c196f90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5628009990 tx=0x7f562800f690 comp rx=0 tx=0).stop 2024-01-24T14:25:19.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.365+0000 7f5642e5e700 1 -- 172.21.15.19:0/2401630042 shutdown_connections 2024-01-24T14:25:19.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.365+0000 7f5642e5e700 1 --2- 172.21.15.19:0/2401630042 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f563c1028f0 0x7f563c19e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:19.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.365+0000 7f5642e5e700 1 --2- 172.21.15.19:0/2401630042 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f563c100d60 0x7f563c196a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:19.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.365+0000 7f5642e5e700 1 --2- 172.21.15.19:0/2401630042 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f562c0a05a0 0x7f562c0a2a60 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:19.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.365+0000 7f5642e5e700 1 --2- 172.21.15.19:0/2401630042 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f563c101f50 0x7f563c196f90 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:19.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.365+0000 7f5642e5e700 1 -- 172.21.15.19:0/2401630042 >> 172.21.15.19:0/2401630042 conn(0x7f563c0fc2b0 msgr2=0x7f563c105440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:19.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.365+0000 7f5642e5e700 1 -- 172.21.15.19:0/2401630042 shutdown_connections 2024-01-24T14:25:19.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:19.365+0000 7f5642e5e700 1 -- 172.21.15.19:0/2401630042 wait complete. 2024-01-24T14:25:19.368 INFO:teuthology.orchestra.run.smithi019.stderr:listed 10 entries 2024-01-24T14:25:19.380 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi019.front.sepia.ceph.com... 2024-01-24T14:25:19.381 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T14:25:19.381 DEBUG:teuthology.orchestra.run.smithi019:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-01-24T14:25:19.414 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi019.stderr:ceph-fuse[95906]: fuse finished with error 0 and tester_r 0 2024-01-24T14:25:19.416 INFO:teuthology.orchestra.run:waiting for 300 2024-01-24T14:25:19.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:19 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/3720100070' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-01-24T14:25:19.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:19 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/3177533848' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-01-24T14:25:19.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:19 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/2401630042' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-01-24T14:25:19.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:19 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/3720100070' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-01-24T14:25:19.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:19 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/3177533848' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-01-24T14:25:19.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:19 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/2401630042' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-01-24T14:25:19.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:19 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/3720100070' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-01-24T14:25:19.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:19 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/3177533848' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-01-24T14:25:19.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:19 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/2401630042' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-01-24T14:25:20.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:20 smithi019 ceph-mon[40663]: pgmap v230: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 1022 B/s wr, 2 op/s 2024-01-24T14:25:20.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:20 smithi123 ceph-mon[44966]: pgmap v230: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 1022 B/s wr, 2 op/s 2024-01-24T14:25:20.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:20 smithi194 ceph-mon[45450]: pgmap v230: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 1022 B/s wr, 2 op/s 2024-01-24T14:25:22.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:22 smithi019 ceph-mon[40663]: pgmap v231: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-01-24T14:25:22.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:22 smithi123 ceph-mon[44966]: pgmap v231: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-01-24T14:25:22.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:22 smithi194 ceph-mon[45450]: pgmap v231: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-01-24T14:25:24.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:24 smithi019 ceph-mon[40663]: pgmap v232: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-01-24T14:25:24.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:24 smithi123 ceph-mon[44966]: pgmap v232: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-01-24T14:25:24.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:24 smithi194 ceph-mon[45450]: pgmap v232: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-01-24T14:25:25.518 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi019.front.sepia.ceph.com 2024-01-24T14:25:25.519 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-24T14:25:25.519 DEBUG:teuthology.orchestra.run.smithi019:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-01-24T14:25:25.548 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T14:25:25.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.736+0000 7f69f2bd4700 1 -- 172.21.15.19:0/1104099275 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f69ec106ac0 msgr2=0x7f69ec108eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:25.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.736+0000 7f69f2bd4700 1 --2- 172.21.15.19:0/1104099275 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f69ec106ac0 0x7f69ec108eb0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f69e400b0a0 tx=0x7f69e400b3b0 comp rx=0 tx=0).stop 2024-01-24T14:25:25.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.736+0000 7f69f2bd4700 1 -- 172.21.15.19:0/1104099275 shutdown_connections 2024-01-24T14:25:25.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.736+0000 7f69f2bd4700 1 --2- 172.21.15.19:0/1104099275 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f69ec106ac0 0x7f69ec108eb0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:25.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.736+0000 7f69f2bd4700 1 --2- 172.21.15.19:0/1104099275 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69ec101860 0x7f69ec103c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:25.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.736+0000 7f69f2bd4700 1 --2- 172.21.15.19:0/1104099275 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f69ec104190 0x7f69ec106580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:25.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.736+0000 7f69f2bd4700 1 -- 172.21.15.19:0/1104099275 >> 172.21.15.19:0/1104099275 conn(0x7f69ec0fb170 msgr2=0x7f69ec0fd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:25.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.736+0000 7f69f2bd4700 1 -- 172.21.15.19:0/1104099275 shutdown_connections 2024-01-24T14:25:25.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.736+0000 7f69f2bd4700 1 -- 172.21.15.19:0/1104099275 wait complete. 2024-01-24T14:25:25.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.737+0000 7f69f2bd4700 1 Processor -- start 2024-01-24T14:25:25.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.737+0000 7f69f2bd4700 1 -- start start 2024-01-24T14:25:25.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.737+0000 7f69f2bd4700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69ec101860 0x7f69ec198d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:25.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.737+0000 7f69f2bd4700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f69ec104190 0x7f69ec199250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:25.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.737+0000 7f69f2bd4700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f69ec106ac0 0x7f69ec1a0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:25.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.737+0000 7f69f2bd4700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69ec1a0850 con 0x7f69ec106ac0 2024-01-24T14:25:25.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.737+0000 7f69f2bd4700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69ec1a09c0 con 0x7f69ec101860 2024-01-24T14:25:25.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.737+0000 7f69f2bd4700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69ec1a0b80 con 0x7f69ec104190 2024-01-24T14:25:25.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.737+0000 7f69f1171700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f69ec106ac0 0x7f69ec1a0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:25.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.737+0000 7f69f1171700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f69ec106ac0 0x7f69ec1a0280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:58798/0 (socket says 172.21.15.19:58798) 2024-01-24T14:25:25.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.737+0000 7f69f1171700 1 -- 172.21.15.19:0/188114763 learned_addr learned my addr 172.21.15.19:0/188114763 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:25:25.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.737+0000 7f69f0970700 1 --2- 172.21.15.19:0/188114763 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69ec101860 0x7f69ec198d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:25.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.737+0000 7f69ebfff700 1 --2- 172.21.15.19:0/188114763 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f69ec104190 0x7f69ec199250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:25.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.738+0000 7f69f1171700 1 -- 172.21.15.19:0/188114763 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f69ec104190 msgr2=0x7f69ec199250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:25.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.738+0000 7f69f1171700 1 --2- 172.21.15.19:0/188114763 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f69ec104190 0x7f69ec199250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:25.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.738+0000 7f69f1171700 1 -- 172.21.15.19:0/188114763 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69ec101860 msgr2=0x7f69ec198d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:25.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.738+0000 7f69f1171700 1 --2- 172.21.15.19:0/188114763 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69ec101860 0x7f69ec198d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:25.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.738+0000 7f69f1171700 1 -- 172.21.15.19:0/188114763 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e4009d00 con 0x7f69ec106ac0 2024-01-24T14:25:25.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.738+0000 7f69f0970700 1 --2- 172.21.15.19:0/188114763 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69ec101860 0x7f69ec198d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T14:25:25.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.738+0000 7f69ebfff700 1 --2- 172.21.15.19:0/188114763 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f69ec104190 0x7f69ec199250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T14:25:25.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.738+0000 7f69f1171700 1 --2- 172.21.15.19:0/188114763 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f69ec106ac0 0x7f69ec1a0280 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f69e4009fd0 tx=0x7f69e400bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:25.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.738+0000 7f69e9ffb700 1 -- 172.21.15.19:0/188114763 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e400e040 con 0x7f69ec106ac0 2024-01-24T14:25:25.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.738+0000 7f69e9ffb700 1 -- 172.21.15.19:0/188114763 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f69e4003e10 con 0x7f69ec106ac0 2024-01-24T14:25:25.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.738+0000 7f69f2bd4700 1 -- 172.21.15.19:0/188114763 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69ec1a0e00 con 0x7f69ec106ac0 2024-01-24T14:25:25.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.738+0000 7f69e9ffb700 1 -- 172.21.15.19:0/188114763 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e401b7e0 con 0x7f69ec106ac0 2024-01-24T14:25:25.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.738+0000 7f69f2bd4700 1 -- 172.21.15.19:0/188114763 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69ec1a11d0 con 0x7f69ec106ac0 2024-01-24T14:25:25.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.739+0000 7f69f2bd4700 1 -- 172.21.15.19:0/188114763 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69ec108640 con 0x7f69ec106ac0 2024-01-24T14:25:25.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.740+0000 7f69e9ffb700 1 -- 172.21.15.19:0/188114763 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f69e401da30 con 0x7f69ec106ac0 2024-01-24T14:25:25.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.741+0000 7f69e9ffb700 1 --2- 172.21.15.19:0/188114763 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f69dc0a0570 0x7f69dc0a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:25.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.742+0000 7f69f0970700 1 --2- 172.21.15.19:0/188114763 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f69dc0a0570 0x7f69dc0a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:25.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.742+0000 7f69e9ffb700 1 -- 172.21.15.19:0/188114763 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f69e40cd4d0 con 0x7f69ec106ac0 2024-01-24T14:25:25.744 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.742+0000 7f69f0970700 1 --2- 172.21.15.19:0/188114763 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f69dc0a0570 0x7f69dc0a2a30 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f69ec19a1e0 tx=0x7f69e0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:25.746 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.745+0000 7f69e9ffb700 1 -- 172.21.15.19:0/188114763 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f69e4096ef0 con 0x7f69ec106ac0 2024-01-24T14:25:25.958 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.957+0000 7f69f2bd4700 1 -- 172.21.15.19:0/188114763 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f69ec02cec0 con 0x7f69ec106ac0 2024-01-24T14:25:25.960 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.959+0000 7f69e9ffb700 1 -- 172.21.15.19:0/188114763 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+23997 (secure 0 0 0) 0x7f69e409a510 con 0x7f69ec106ac0 2024-01-24T14:25:25.961 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:25:25.961 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":82,"fsid":"654b0c8e-bac3-11ee-95b1-87774f69a715","created":"2024-01-24T14:19:10.699369+0000","modified":"2024-01-24T14:25:07.960902+0000","last_up_change":"2024-01-24T14:24:30.032711+0000","last_in_change":"2024-01-24T14:24:13.776095+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T14:21:53.453429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-24T14:25:01.967538+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-24T14:25:02.864396+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-01-24T14:25:05.950591+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"81380ec6-8e19-4eb4-8158-8fafaed8d7e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":2456786579},{"type":"v1","addr":"172.21.15.19:6803","nonce":2456786579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":2456786579},{"type":"v1","addr":"172.21.15.19:6805","nonce":2456786579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":2456786579},{"type":"v1","addr":"172.21.15.19:6809","nonce":2456786579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":2456786579},{"type":"v1","addr":"172.21.15.19:6807","nonce":2456786579}]},"public_addr":"172.21.15.19:6803/2456786579","cluster_addr":"172.21.15.19:6805/2456786579","heartbeat_back_addr":"172.21.15.19:6809/2456786579","heartbeat_front_addr":"172.21.15.19:6807/2456786579","state":["exists","up"]},{"osd":1,"uuid":"687f5036-a6d2-4aa1-86e7-7570e51ef179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":2407548889},{"type":"v1","addr":"172.21.15.123:6801","nonce":2407548889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":2407548889},{"type":"v1","addr":"172.21.15.123:6803","nonce":2407548889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":2407548889},{"type":"v1","addr":"172.21.15.123:6807","nonce":2407548889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":2407548889},{"type":"v1","addr":"172.21.15.123:6805","nonce":2407548889}]},"public_addr":"172.21.15.123:6801/2407548889","cluster_addr":"172.21.15.123:6803/2407548889","heartbeat_back_addr":"172.21.15.123:6807/2407548889","heartbeat_front_addr":"172.21.15.123:6805/2407548889","state":["exists","up"]},{"osd":2,"uuid":"3e810904-371d-4ca8-b43c-434957a2aa12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":3632698637},{"type":"v1","addr":"172.21.15.194:6801","nonce":3632698637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":3632698637},{"type":"v1","addr":"172.21.15.194:6803","nonce":3632698637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":3632698637},{"type":"v1","addr":"172.21.15.194:6807","nonce":3632698637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":3632698637},{"type":"v1","addr":"172.21.15.194:6805","nonce":3632698637}]},"public_addr":"172.21.15.194:6801/3632698637","cluster_addr":"172.21.15.194:6803/3632698637","heartbeat_back_addr":"172.21.15.194:6807/3632698637","heartbeat_front_addr":"172.21.15.194:6805/3632698637","state":["exists","up"]},{"osd":3,"uuid":"3665d6e0-010a-40bd-b0c6-682af56a0cbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":3868219159},{"type":"v1","addr":"172.21.15.19:6811","nonce":3868219159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3868219159},{"type":"v1","addr":"172.21.15.19:6813","nonce":3868219159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":3868219159},{"type":"v1","addr":"172.21.15.19:6817","nonce":3868219159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":3868219159},{"type":"v1","addr":"172.21.15.19:6815","nonce":3868219159}]},"public_addr":"172.21.15.19:6811/3868219159","cluster_addr":"172.21.15.19:6813/3868219159","heartbeat_back_addr":"172.21.15.19:6817/3868219159","heartbeat_front_addr":"172.21.15.19:6815/3868219159","state":["exists","up"]},{"osd":4,"uuid":"029fb28f-9af5-4239-8ee5-5ac5d04cd34b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":3611579809},{"type":"v1","addr":"172.21.15.123:6809","nonce":3611579809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":3611579809},{"type":"v1","addr":"172.21.15.123:6811","nonce":3611579809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":3611579809},{"type":"v1","addr":"172.21.15.123:6815","nonce":3611579809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":3611579809},{"type":"v1","addr":"172.21.15.123:6813","nonce":3611579809}]},"public_addr":"172.21.15.123:6809/3611579809","cluster_addr":"172.21.15.123:6811/3611579809","heartbeat_back_addr":"172.21.15.123:6815/3611579809","heartbeat_front_addr":"172.21.15.123:6813/3611579809","state":["exists","up"]},{"osd":5,"uuid":"6e97f23b-f803-4fed-a0a2-7aee7de4970f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":2483361871},{"type":"v1","addr":"172.21.15.194:6809","nonce":2483361871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":2483361871},{"type":"v1","addr":"172.21.15.194:6811","nonce":2483361871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":2483361871},{"type":"v1","addr":"172.21.15.194:6815","nonce":2483361871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":2483361871},{"type":"v1","addr":"172.21.15.194:6813","nonce":2483361871}]},"public_addr":"172.21.15.194:6809/2483361871","cluster_addr":"172.21.15.194:6811/2483361871","heartbeat_back_addr":"172.21.15.194:6815/2483361871","heartbeat_front_addr":"172.21.15.194:6813/2483361871","state":["exists","up"]},{"osd":6,"uuid":"c8c8ca2e-4cc6-4861-94b0-6211e4936c25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":1290513386},{"type":"v1","addr":"172.21.15.19:6819","nonce":1290513386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":1290513386},{"type":"v1","addr":"172.21.15.19:6821","nonce":1290513386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":1290513386},{"type":"v1","addr":"172.21.15.19:6825","nonce":1290513386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":1290513386},{"type":"v1","addr":"172.21.15.19:6823","nonce":1290513386}]},"public_addr":"172.21.15.19:6819/1290513386","cluster_addr":"172.21.15.19:6821/1290513386","heartbeat_back_addr":"172.21.15.19:6825/1290513386","heartbeat_front_addr":"172.21.15.19:6823/1290513386","state":["exists","up"]},{"osd":7,"uuid":"9aaea7a8-7bc3-4e32-9328-20a6c30a6d8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1391966446},{"type":"v1","addr":"172.21.15.123:6817","nonce":1391966446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1391966446},{"type":"v1","addr":"172.21.15.123:6819","nonce":1391966446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1391966446},{"type":"v1","addr":"172.21.15.123:6823","nonce":1391966446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1391966446},{"type":"v1","addr":"172.21.15.123:6821","nonce":1391966446}]},"public_addr":"172.21.15.123:6817/1391966446","cluster_addr":"172.21.15.123:6819/1391966446","heartbeat_back_addr":"172.21.15.123:6823/1391966446","heartbeat_front_addr":"172.21.15.123:6821/1391966446","state":["exists","up"]},{"osd":8,"uuid":"e918ec1b-7608-4f73-9067-29082ea63999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":2260577139},{"type":"v1","addr":"172.21.15.194:6817","nonce":2260577139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":2260577139},{"type":"v1","addr":"172.21.15.194:6819","nonce":2260577139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":2260577139},{"type":"v1","addr":"172.21.15.194:6823","nonce":2260577139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":2260577139},{"type":"v1","addr":"172.21.15.194:6821","nonce":2260577139}]},"public_addr":"172.21.15.194:6817/2260577139","cluster_addr":"172.21.15.194:6819/2260577139","heartbeat_back_addr":"172.21.15.194:6823/2260577139","heartbeat_front_addr":"172.21.15.194:6821/2260577139","state":["exists","up"]},{"osd":9,"uuid":"0fd76722-3319-4828-ab68-108607a4bd0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":4004843075},{"type":"v1","addr":"172.21.15.19:6827","nonce":4004843075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":4004843075},{"type":"v1","addr":"172.21.15.19:6829","nonce":4004843075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":4004843075},{"type":"v1","addr":"172.21.15.19:6833","nonce":4004843075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":4004843075},{"type":"v1","addr":"172.21.15.19:6831","nonce":4004843075}]},"public_addr":"172.21.15.19:6827/4004843075","cluster_addr":"172.21.15.19:6829/4004843075","heartbeat_back_addr":"172.21.15.19:6833/4004843075","heartbeat_front_addr":"172.21.15.19:6831/4004843075","state":["exists","up"]},{"osd":10,"uuid":"624bb8f5-6d72-43d5-9986-1b48b8a564ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2591845068},{"type":"v1","addr":"172.21.15.123:6825","nonce":2591845068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":2591845068},{"type":"v1","addr":"172.21.15.123:6827","nonce":2591845068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2591845068},{"type":"v1","addr":"172.21.15.123:6831","nonce":2591845068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2591845068},{"type":"v1","addr":"172.21.15.123:6829","nonce":2591845068}]},"public_addr":"172.21.15.123:6825/2591845068","cluster_addr":"172.21.15.123:6827/2591845068","heartbeat_back_addr":"172.21.15.123:6831/2591845068","heartbeat_front_addr":"172.21.15.123:6829/2591845068","state":["exists","up"]},{"osd":11,"uuid":"219a55b3-b01e-471d-90da-1a3043bf3e34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":3985496260},{"type":"v1","addr":"172.21.15.194:6825","nonce":3985496260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":3985496260},{"type":"v1","addr":"172.21.15.194:6827","nonce":3985496260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":3985496260},{"type":"v1","addr":"172.21.15.194:6831","nonce":3985496260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":3985496260},{"type":"v1","addr":"172.21.15.194:6829","nonce":3985496260}]},"public_addr":"172.21.15.194:6825/3985496260","cluster_addr":"172.21.15.194:6827/3985496260","heartbeat_back_addr":"172.21.15.194:6831/3985496260","heartbeat_front_addr":"172.21.15.194:6829/3985496260","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:21:17.278347+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:21:34.469282+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:21:51.139919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:22:09.126458+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:22:26.186801+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:22:43.178997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:23:01.109570+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:23:18.592191+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:23:35.685046+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:23:53.574820+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:24:10.987531+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:24:28.231206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.19:0/1047252857":"2024-01-25T14:20:02.911218+0000","172.21.15.19:6801/3758061162":"2024-01-25T14:20:02.911218+0000","172.21.15.19:6800/3758061162":"2024-01-25T14:20:02.911218+0000","172.21.15.19:6801/2754937713":"2024-01-25T14:19:39.853212+0000","172.21.15.19:0/4062420470":"2024-01-25T14:19:39.853212+0000","172.21.15.19:6800/2754937713":"2024-01-25T14:19:39.853212+0000","172.21.15.19:0/822613692":"2024-01-25T14:20:02.911218+0000","172.21.15.19:0/3970666946":"2024-01-25T14:19:39.853212+0000","172.21.15.19:0/1116053938":"2024-01-25T14:19:39.853212+0000","172.21.15.19:0/1456770388":"2024-01-25T14:20:02.911218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T14:25:25.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.961+0000 7f69f2bd4700 1 -- 172.21.15.19:0/188114763 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f69dc0a0570 msgr2=0x7f69dc0a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:25.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.961+0000 7f69f2bd4700 1 --2- 172.21.15.19:0/188114763 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f69dc0a0570 0x7f69dc0a2a30 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f69ec19a1e0 tx=0x7f69e0009260 comp rx=0 tx=0).stop 2024-01-24T14:25:25.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.961+0000 7f69f2bd4700 1 -- 172.21.15.19:0/188114763 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f69ec106ac0 msgr2=0x7f69ec1a0280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:25.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.961+0000 7f69f2bd4700 1 --2- 172.21.15.19:0/188114763 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f69ec106ac0 0x7f69ec1a0280 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f69e4009fd0 tx=0x7f69e400bd90 comp rx=0 tx=0).stop 2024-01-24T14:25:25.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.961+0000 7f69f2bd4700 1 -- 172.21.15.19:0/188114763 shutdown_connections 2024-01-24T14:25:25.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.961+0000 7f69f2bd4700 1 --2- 172.21.15.19:0/188114763 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f69dc0a0570 0x7f69dc0a2a30 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:25.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.961+0000 7f69f2bd4700 1 --2- 172.21.15.19:0/188114763 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f69ec106ac0 0x7f69ec1a0280 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:25.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.961+0000 7f69f2bd4700 1 --2- 172.21.15.19:0/188114763 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69ec101860 0x7f69ec198d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:25.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.961+0000 7f69f2bd4700 1 --2- 172.21.15.19:0/188114763 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f69ec104190 0x7f69ec199250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:25.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.962+0000 7f69f2bd4700 1 -- 172.21.15.19:0/188114763 >> 172.21.15.19:0/188114763 conn(0x7f69ec0fb170 msgr2=0x7f69ec0ffe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:25.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.962+0000 7f69f2bd4700 1 -- 172.21.15.19:0/188114763 shutdown_connections 2024-01-24T14:25:25.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:25.962+0000 7f69f2bd4700 1 -- 172.21.15.19:0/188114763 wait complete. 2024-01-24T14:25:25.977 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-24T14:25:26.117 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.114+0000 7f7702cad700 1 -- 172.21.15.19:0/1891479303 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f76fc101f30 msgr2=0x7f76fc102390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:26.117 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.114+0000 7f7702cad700 1 --2- 172.21.15.19:0/1891479303 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f76fc101f30 0x7f76fc102390 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f76f00099c0 tx=0x7f76f0009cd0 comp rx=0 tx=0).stop 2024-01-24T14:25:26.117 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.114+0000 7f7702cad700 1 -- 172.21.15.19:0/1891479303 shutdown_connections 2024-01-24T14:25:26.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.114+0000 7f7702cad700 1 --2- 172.21.15.19:0/1891479303 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76fc1028d0 0x7f76fc109180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:26.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.114+0000 7f7702cad700 1 --2- 172.21.15.19:0/1891479303 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76fc100d40 0x7f76fc101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:26.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.114+0000 7f7702cad700 1 --2- 172.21.15.19:0/1891479303 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f76fc101f30 0x7f76fc102390 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:26.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.114+0000 7f7702cad700 1 -- 172.21.15.19:0/1891479303 >> 172.21.15.19:0/1891479303 conn(0x7f76fc0fc290 msgr2=0x7f76fc0fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:26.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.114+0000 7f7702cad700 1 -- 172.21.15.19:0/1891479303 shutdown_connections 2024-01-24T14:25:26.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.114+0000 7f7702cad700 1 -- 172.21.15.19:0/1891479303 wait complete. 2024-01-24T14:25:26.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.115+0000 7f7702cad700 1 Processor -- start 2024-01-24T14:25:26.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.115+0000 7f7702cad700 1 -- start start 2024-01-24T14:25:26.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.115+0000 7f7702cad700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76fc100d40 0x7f76fc074af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:26.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.115+0000 7f7702cad700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76fc101f30 0x7f76fc073130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:26.118 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.115+0000 7f7702cad700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f76fc1028d0 0x7f76fc073670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:26.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.115+0000 7f7702cad700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76fc073e10 con 0x7f76fc1028d0 2024-01-24T14:25:26.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.115+0000 7f7702cad700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76fc073f80 con 0x7f76fc101f30 2024-01-24T14:25:26.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.115+0000 7f7702cad700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76fc1a0990 con 0x7f76fc100d40 2024-01-24T14:25:26.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.115+0000 7f770124a700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f76fc1028d0 0x7f76fc073670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:26.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.115+0000 7f770124a700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f76fc1028d0 0x7f76fc073670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:58814/0 (socket says 172.21.15.19:58814) 2024-01-24T14:25:26.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.115+0000 7f770124a700 1 -- 172.21.15.19:0/3510963862 learned_addr learned my addr 172.21.15.19:0/3510963862 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:25:26.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.115+0000 7f76fbfff700 1 --2- 172.21.15.19:0/3510963862 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76fc101f30 0x7f76fc073130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:26.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.115+0000 7f7700a49700 1 --2- 172.21.15.19:0/3510963862 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76fc100d40 0x7f76fc074af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:26.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.115+0000 7f770124a700 1 -- 172.21.15.19:0/3510963862 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76fc100d40 msgr2=0x7f76fc074af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:26.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.115+0000 7f770124a700 1 --2- 172.21.15.19:0/3510963862 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76fc100d40 0x7f76fc074af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:26.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.115+0000 7f770124a700 1 -- 172.21.15.19:0/3510963862 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76fc101f30 msgr2=0x7f76fc073130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:26.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.115+0000 7f770124a700 1 --2- 172.21.15.19:0/3510963862 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76fc101f30 0x7f76fc073130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:26.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.115+0000 7f770124a700 1 -- 172.21.15.19:0/3510963862 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76f00095f0 con 0x7f76fc1028d0 2024-01-24T14:25:26.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.116+0000 7f770124a700 1 --2- 172.21.15.19:0/3510963862 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f76fc1028d0 0x7f76fc073670 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f76f4007d70 tx=0x7f76f400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:26.119 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.116+0000 7f76f9ffb700 1 -- 172.21.15.19:0/3510963862 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f400ac90 con 0x7f76fc1028d0 2024-01-24T14:25:26.120 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.116+0000 7f76f9ffb700 1 -- 172.21.15.19:0/3510963862 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f76f400adf0 con 0x7f76fc1028d0 2024-01-24T14:25:26.120 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.116+0000 7f7702cad700 1 -- 172.21.15.19:0/3510963862 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76fc1a0c70 con 0x7f76fc1028d0 2024-01-24T14:25:26.120 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.116+0000 7f76f9ffb700 1 -- 172.21.15.19:0/3510963862 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f40115e0 con 0x7f76fc1028d0 2024-01-24T14:25:26.120 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.116+0000 7f7702cad700 1 -- 172.21.15.19:0/3510963862 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76fc1a1130 con 0x7f76fc1028d0 2024-01-24T14:25:26.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.117+0000 7f7702cad700 1 -- 172.21.15.19:0/3510963862 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76fc1081e0 con 0x7f76fc1028d0 2024-01-24T14:25:26.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.118+0000 7f76f9ffb700 1 -- 172.21.15.19:0/3510963862 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f76f4012420 con 0x7f76fc1028d0 2024-01-24T14:25:26.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.119+0000 7f76f9ffb700 1 --2- 172.21.15.19:0/3510963862 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f76e40a0570 0x7f76e40a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:26.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.119+0000 7f7700a49700 1 --2- 172.21.15.19:0/3510963862 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f76e40a0570 0x7f76e40a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:26.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.119+0000 7f76f9ffb700 1 -- 172.21.15.19:0/3510963862 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f76f40cbdb0 con 0x7f76fc1028d0 2024-01-24T14:25:26.121 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.119+0000 7f7700a49700 1 --2- 172.21.15.19:0/3510963862 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f76e40a0570 0x7f76e40a2a30 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f76fc101d90 tx=0x7f76ec00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:26.124 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.123+0000 7f76f9ffb700 1 -- 172.21.15.19:0/3510963862 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f76f40d0070 con 0x7f76fc1028d0 2024-01-24T14:25:26.341 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.340+0000 7f7702cad700 1 -- 172.21.15.19:0/3510963862 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f76fc02cc70 con 0x7f76fc1028d0 2024-01-24T14:25:26.342 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.342+0000 7f76f9ffb700 1 -- 172.21.15.19:0/3510963862 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+23997 (secure 0 0 0) 0x7f76f4095700 con 0x7f76fc1028d0 2024-01-24T14:25:26.343 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-01-24T14:25:26.343 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":82,"fsid":"654b0c8e-bac3-11ee-95b1-87774f69a715","created":"2024-01-24T14:19:10.699369+0000","modified":"2024-01-24T14:25:07.960902+0000","last_up_change":"2024-01-24T14:24:30.032711+0000","last_in_change":"2024-01-24T14:24:13.776095+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-24T14:21:53.453429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-24T14:25:01.967538+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-24T14:25:02.864396+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"78","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-01-24T14:25:05.950591+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"81380ec6-8e19-4eb4-8158-8fafaed8d7e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":2456786579},{"type":"v1","addr":"172.21.15.19:6803","nonce":2456786579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":2456786579},{"type":"v1","addr":"172.21.15.19:6805","nonce":2456786579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":2456786579},{"type":"v1","addr":"172.21.15.19:6809","nonce":2456786579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":2456786579},{"type":"v1","addr":"172.21.15.19:6807","nonce":2456786579}]},"public_addr":"172.21.15.19:6803/2456786579","cluster_addr":"172.21.15.19:6805/2456786579","heartbeat_back_addr":"172.21.15.19:6809/2456786579","heartbeat_front_addr":"172.21.15.19:6807/2456786579","state":["exists","up"]},{"osd":1,"uuid":"687f5036-a6d2-4aa1-86e7-7570e51ef179","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":2407548889},{"type":"v1","addr":"172.21.15.123:6801","nonce":2407548889}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":2407548889},{"type":"v1","addr":"172.21.15.123:6803","nonce":2407548889}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":2407548889},{"type":"v1","addr":"172.21.15.123:6807","nonce":2407548889}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":2407548889},{"type":"v1","addr":"172.21.15.123:6805","nonce":2407548889}]},"public_addr":"172.21.15.123:6801/2407548889","cluster_addr":"172.21.15.123:6803/2407548889","heartbeat_back_addr":"172.21.15.123:6807/2407548889","heartbeat_front_addr":"172.21.15.123:6805/2407548889","state":["exists","up"]},{"osd":2,"uuid":"3e810904-371d-4ca8-b43c-434957a2aa12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":3632698637},{"type":"v1","addr":"172.21.15.194:6801","nonce":3632698637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":3632698637},{"type":"v1","addr":"172.21.15.194:6803","nonce":3632698637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":3632698637},{"type":"v1","addr":"172.21.15.194:6807","nonce":3632698637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":3632698637},{"type":"v1","addr":"172.21.15.194:6805","nonce":3632698637}]},"public_addr":"172.21.15.194:6801/3632698637","cluster_addr":"172.21.15.194:6803/3632698637","heartbeat_back_addr":"172.21.15.194:6807/3632698637","heartbeat_front_addr":"172.21.15.194:6805/3632698637","state":["exists","up"]},{"osd":3,"uuid":"3665d6e0-010a-40bd-b0c6-682af56a0cbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":3868219159},{"type":"v1","addr":"172.21.15.19:6811","nonce":3868219159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":3868219159},{"type":"v1","addr":"172.21.15.19:6813","nonce":3868219159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":3868219159},{"type":"v1","addr":"172.21.15.19:6817","nonce":3868219159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":3868219159},{"type":"v1","addr":"172.21.15.19:6815","nonce":3868219159}]},"public_addr":"172.21.15.19:6811/3868219159","cluster_addr":"172.21.15.19:6813/3868219159","heartbeat_back_addr":"172.21.15.19:6817/3868219159","heartbeat_front_addr":"172.21.15.19:6815/3868219159","state":["exists","up"]},{"osd":4,"uuid":"029fb28f-9af5-4239-8ee5-5ac5d04cd34b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":3611579809},{"type":"v1","addr":"172.21.15.123:6809","nonce":3611579809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":3611579809},{"type":"v1","addr":"172.21.15.123:6811","nonce":3611579809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":3611579809},{"type":"v1","addr":"172.21.15.123:6815","nonce":3611579809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":3611579809},{"type":"v1","addr":"172.21.15.123:6813","nonce":3611579809}]},"public_addr":"172.21.15.123:6809/3611579809","cluster_addr":"172.21.15.123:6811/3611579809","heartbeat_back_addr":"172.21.15.123:6815/3611579809","heartbeat_front_addr":"172.21.15.123:6813/3611579809","state":["exists","up"]},{"osd":5,"uuid":"6e97f23b-f803-4fed-a0a2-7aee7de4970f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":2483361871},{"type":"v1","addr":"172.21.15.194:6809","nonce":2483361871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":2483361871},{"type":"v1","addr":"172.21.15.194:6811","nonce":2483361871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":2483361871},{"type":"v1","addr":"172.21.15.194:6815","nonce":2483361871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":2483361871},{"type":"v1","addr":"172.21.15.194:6813","nonce":2483361871}]},"public_addr":"172.21.15.194:6809/2483361871","cluster_addr":"172.21.15.194:6811/2483361871","heartbeat_back_addr":"172.21.15.194:6815/2483361871","heartbeat_front_addr":"172.21.15.194:6813/2483361871","state":["exists","up"]},{"osd":6,"uuid":"c8c8ca2e-4cc6-4861-94b0-6211e4936c25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":1290513386},{"type":"v1","addr":"172.21.15.19:6819","nonce":1290513386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6820","nonce":1290513386},{"type":"v1","addr":"172.21.15.19:6821","nonce":1290513386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":1290513386},{"type":"v1","addr":"172.21.15.19:6825","nonce":1290513386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":1290513386},{"type":"v1","addr":"172.21.15.19:6823","nonce":1290513386}]},"public_addr":"172.21.15.19:6819/1290513386","cluster_addr":"172.21.15.19:6821/1290513386","heartbeat_back_addr":"172.21.15.19:6825/1290513386","heartbeat_front_addr":"172.21.15.19:6823/1290513386","state":["exists","up"]},{"osd":7,"uuid":"9aaea7a8-7bc3-4e32-9328-20a6c30a6d8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":1391966446},{"type":"v1","addr":"172.21.15.123:6817","nonce":1391966446}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":1391966446},{"type":"v1","addr":"172.21.15.123:6819","nonce":1391966446}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":1391966446},{"type":"v1","addr":"172.21.15.123:6823","nonce":1391966446}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":1391966446},{"type":"v1","addr":"172.21.15.123:6821","nonce":1391966446}]},"public_addr":"172.21.15.123:6817/1391966446","cluster_addr":"172.21.15.123:6819/1391966446","heartbeat_back_addr":"172.21.15.123:6823/1391966446","heartbeat_front_addr":"172.21.15.123:6821/1391966446","state":["exists","up"]},{"osd":8,"uuid":"e918ec1b-7608-4f73-9067-29082ea63999","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":2260577139},{"type":"v1","addr":"172.21.15.194:6817","nonce":2260577139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":2260577139},{"type":"v1","addr":"172.21.15.194:6819","nonce":2260577139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":2260577139},{"type":"v1","addr":"172.21.15.194:6823","nonce":2260577139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":2260577139},{"type":"v1","addr":"172.21.15.194:6821","nonce":2260577139}]},"public_addr":"172.21.15.194:6817/2260577139","cluster_addr":"172.21.15.194:6819/2260577139","heartbeat_back_addr":"172.21.15.194:6823/2260577139","heartbeat_front_addr":"172.21.15.194:6821/2260577139","state":["exists","up"]},{"osd":9,"uuid":"0fd76722-3319-4828-ab68-108607a4bd0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6826","nonce":4004843075},{"type":"v1","addr":"172.21.15.19:6827","nonce":4004843075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":4004843075},{"type":"v1","addr":"172.21.15.19:6829","nonce":4004843075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6832","nonce":4004843075},{"type":"v1","addr":"172.21.15.19:6833","nonce":4004843075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":4004843075},{"type":"v1","addr":"172.21.15.19:6831","nonce":4004843075}]},"public_addr":"172.21.15.19:6827/4004843075","cluster_addr":"172.21.15.19:6829/4004843075","heartbeat_back_addr":"172.21.15.19:6833/4004843075","heartbeat_front_addr":"172.21.15.19:6831/4004843075","state":["exists","up"]},{"osd":10,"uuid":"624bb8f5-6d72-43d5-9986-1b48b8a564ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":2591845068},{"type":"v1","addr":"172.21.15.123:6825","nonce":2591845068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":2591845068},{"type":"v1","addr":"172.21.15.123:6827","nonce":2591845068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":2591845068},{"type":"v1","addr":"172.21.15.123:6831","nonce":2591845068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":2591845068},{"type":"v1","addr":"172.21.15.123:6829","nonce":2591845068}]},"public_addr":"172.21.15.123:6825/2591845068","cluster_addr":"172.21.15.123:6827/2591845068","heartbeat_back_addr":"172.21.15.123:6831/2591845068","heartbeat_front_addr":"172.21.15.123:6829/2591845068","state":["exists","up"]},{"osd":11,"uuid":"219a55b3-b01e-471d-90da-1a3043bf3e34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":3985496260},{"type":"v1","addr":"172.21.15.194:6825","nonce":3985496260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":3985496260},{"type":"v1","addr":"172.21.15.194:6827","nonce":3985496260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":3985496260},{"type":"v1","addr":"172.21.15.194:6831","nonce":3985496260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":3985496260},{"type":"v1","addr":"172.21.15.194:6829","nonce":3985496260}]},"public_addr":"172.21.15.194:6825/3985496260","cluster_addr":"172.21.15.194:6827/3985496260","heartbeat_back_addr":"172.21.15.194:6831/3985496260","heartbeat_front_addr":"172.21.15.194:6829/3985496260","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:21:17.278347+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:21:34.469282+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:21:51.139919+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:22:09.126458+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:22:26.186801+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:22:43.178997+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:23:01.109570+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:23:18.592191+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:23:35.685046+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:23:53.574820+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:24:10.987531+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-24T14:24:28.231206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.19:0/1047252857":"2024-01-25T14:20:02.911218+0000","172.21.15.19:6801/3758061162":"2024-01-25T14:20:02.911218+0000","172.21.15.19:6800/3758061162":"2024-01-25T14:20:02.911218+0000","172.21.15.19:6801/2754937713":"2024-01-25T14:19:39.853212+0000","172.21.15.19:0/4062420470":"2024-01-25T14:19:39.853212+0000","172.21.15.19:6800/2754937713":"2024-01-25T14:19:39.853212+0000","172.21.15.19:0/822613692":"2024-01-25T14:20:02.911218+0000","172.21.15.19:0/3970666946":"2024-01-25T14:19:39.853212+0000","172.21.15.19:0/1116053938":"2024-01-25T14:19:39.853212+0000","172.21.15.19:0/1456770388":"2024-01-25T14:20:02.911218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-24T14:25:26.345 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.344+0000 7f7702cad700 1 -- 172.21.15.19:0/3510963862 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f76e40a0570 msgr2=0x7f76e40a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:26.345 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.344+0000 7f7702cad700 1 --2- 172.21.15.19:0/3510963862 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f76e40a0570 0x7f76e40a2a30 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f76fc101d90 tx=0x7f76ec00b410 comp rx=0 tx=0).stop 2024-01-24T14:25:26.345 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.344+0000 7f7702cad700 1 -- 172.21.15.19:0/3510963862 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f76fc1028d0 msgr2=0x7f76fc073670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:26.345 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.344+0000 7f7702cad700 1 --2- 172.21.15.19:0/3510963862 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f76fc1028d0 0x7f76fc073670 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f76f4007d70 tx=0x7f76f400a340 comp rx=0 tx=0).stop 2024-01-24T14:25:26.345 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.344+0000 7f7702cad700 1 -- 172.21.15.19:0/3510963862 shutdown_connections 2024-01-24T14:25:26.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.344+0000 7f7702cad700 1 --2- 172.21.15.19:0/3510963862 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f76e40a0570 0x7f76e40a2a30 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:26.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.344+0000 7f7702cad700 1 --2- 172.21.15.19:0/3510963862 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f76fc1028d0 0x7f76fc073670 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:26.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.344+0000 7f7702cad700 1 --2- 172.21.15.19:0/3510963862 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76fc100d40 0x7f76fc074af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:26.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.344+0000 7f7702cad700 1 --2- 172.21.15.19:0/3510963862 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76fc101f30 0x7f76fc073130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:26.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.344+0000 7f7702cad700 1 -- 172.21.15.19:0/3510963862 >> 172.21.15.19:0/3510963862 conn(0x7f76fc0fc290 msgr2=0x7f76fc105320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:26.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.344+0000 7f7702cad700 1 -- 172.21.15.19:0/3510963862 shutdown_connections 2024-01-24T14:25:26.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.344+0000 7f7702cad700 1 -- 172.21.15.19:0/3510963862 wait complete. 2024-01-24T14:25:26.360 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-24T14:25:26.506 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.500+0000 7f4e7c78e700 1 -- 172.21.15.19:0/3586115215 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4e740ffca0 msgr2=0x7f4e741000c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:26.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.500+0000 7f4e7c78e700 1 --2- 172.21.15.19:0/3586115215 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4e740ffca0 0x7f4e741000c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4e700099c0 tx=0x7f4e70009cd0 comp rx=0 tx=0).stop 2024-01-24T14:25:26.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.500+0000 7f4e7c78e700 1 -- 172.21.15.19:0/3586115215 shutdown_connections 2024-01-24T14:25:26.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.500+0000 7f4e7c78e700 1 --2- 172.21.15.19:0/3586115215 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4e740ffca0 0x7f4e741000c0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:26.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.500+0000 7f4e7c78e700 1 --2- 172.21.15.19:0/3586115215 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e74101830 0x7f4e741080e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:26.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.500+0000 7f4e7c78e700 1 --2- 172.21.15.19:0/3586115215 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e74100e90 0x7f4e741012f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:26.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.500+0000 7f4e7c78e700 1 -- 172.21.15.19:0/3586115215 >> 172.21.15.19:0/3586115215 conn(0x7f4e740fb210 msgr2=0x7f4e740fd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:26.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.500+0000 7f4e7c78e700 1 -- 172.21.15.19:0/3586115215 shutdown_connections 2024-01-24T14:25:26.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.500+0000 7f4e7c78e700 1 -- 172.21.15.19:0/3586115215 wait complete. 2024-01-24T14:25:26.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.500+0000 7f4e7c78e700 1 Processor -- start 2024-01-24T14:25:26.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.501+0000 7f4e7c78e700 1 -- start start 2024-01-24T14:25:26.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.501+0000 7f4e7c78e700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e740ffca0 0x7f4e74069be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:26.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.501+0000 7f4e7c78e700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4e74100e90 0x7f4e7419b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:26.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.501+0000 7f4e7c78e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e74101830 0x7f4e7419dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:26.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.501+0000 7f4e7c78e700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7419e570 con 0x7f4e74100e90 2024-01-24T14:25:26.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.501+0000 7f4e7c78e700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7419e6e0 con 0x7f4e740ffca0 2024-01-24T14:25:26.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.501+0000 7f4e7c78e700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7419e8a0 con 0x7f4e74101830 2024-01-24T14:25:26.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.501+0000 7f4e79d29700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4e74100e90 0x7f4e7419b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:26.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.501+0000 7f4e79d29700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4e74100e90 0x7f4e7419b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:58834/0 (socket says 172.21.15.19:58834) 2024-01-24T14:25:26.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.501+0000 7f4e79d29700 1 -- 172.21.15.19:0/3035990576 learned_addr learned my addr 172.21.15.19:0/3035990576 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:25:26.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.501+0000 7f4e7ad2b700 1 --2- 172.21.15.19:0/3035990576 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e74101830 0x7f4e7419dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:26.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.501+0000 7f4e7a52a700 1 --2- 172.21.15.19:0/3035990576 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e740ffca0 0x7f4e74069be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:26.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.501+0000 7f4e79d29700 1 -- 172.21.15.19:0/3035990576 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e74101830 msgr2=0x7f4e7419dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:26.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.501+0000 7f4e79d29700 1 --2- 172.21.15.19:0/3035990576 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e74101830 0x7f4e7419dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:26.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.501+0000 7f4e79d29700 1 -- 172.21.15.19:0/3035990576 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e740ffca0 msgr2=0x7f4e74069be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:26.509 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.501+0000 7f4e79d29700 1 --2- 172.21.15.19:0/3035990576 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e740ffca0 0x7f4e74069be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:26.509 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.501+0000 7f4e79d29700 1 -- 172.21.15.19:0/3035990576 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e700095f0 con 0x7f4e74100e90 2024-01-24T14:25:26.509 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.501+0000 7f4e79d29700 1 --2- 172.21.15.19:0/3035990576 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4e74100e90 0x7f4e7419b750 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4e6400d980 tx=0x7f4e6400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:26.509 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.502+0000 7f4e6b7fe700 1 -- 172.21.15.19:0/3035990576 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e64004220 con 0x7f4e74100e90 2024-01-24T14:25:26.509 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.502+0000 7f4e6b7fe700 1 -- 172.21.15.19:0/3035990576 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4e64007640 con 0x7f4e74100e90 2024-01-24T14:25:26.509 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.502+0000 7f4e6b7fe700 1 -- 172.21.15.19:0/3035990576 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e64005690 con 0x7f4e74100e90 2024-01-24T14:25:26.509 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.502+0000 7f4e7c78e700 1 -- 172.21.15.19:0/3035990576 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e7419eb80 con 0x7f4e74100e90 2024-01-24T14:25:26.509 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.502+0000 7f4e7c78e700 1 -- 172.21.15.19:0/3035990576 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e7419f090 con 0x7f4e74100e90 2024-01-24T14:25:26.509 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.503+0000 7f4e7c78e700 1 -- 172.21.15.19:0/3035990576 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e74106c20 con 0x7f4e74100e90 2024-01-24T14:25:26.509 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.504+0000 7f4e6b7fe700 1 -- 172.21.15.19:0/3035990576 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f4e64004750 con 0x7f4e74100e90 2024-01-24T14:25:26.509 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.505+0000 7f4e6b7fe700 1 --2- 172.21.15.19:0/3035990576 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f4e600a05a0 0x7f4e600a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:26.509 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.505+0000 7f4e6b7fe700 1 -- 172.21.15.19:0/3035990576 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f4e640cbe00 con 0x7f4e74100e90 2024-01-24T14:25:26.509 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.505+0000 7f4e7a52a700 1 --2- 172.21.15.19:0/3035990576 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f4e600a05a0 0x7f4e600a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:26.509 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.505+0000 7f4e7a52a700 1 --2- 172.21.15.19:0/3035990576 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f4e600a05a0 0x7f4e600a2a60 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4e70009f50 tx=0x7f4e7000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:26.510 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.508+0000 7f4e6b7fe700 1 -- 172.21.15.19:0/3035990576 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f4e640957b0 con 0x7f4e74100e90 2024-01-24T14:25:26.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:26 smithi019 ceph-mon[40663]: pgmap v233: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-01-24T14:25:26.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:26 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/188114763' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T14:25:26.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:26 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/3510963862' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T14:25:26.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:26.773+0000 7f4e7c78e700 1 -- 172.21.15.19:0/3035990576 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f4e7402cc70 con 0x7f4e74100e90 2024-01-24T14:25:26.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:26 smithi123 ceph-mon[44966]: pgmap v233: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-01-24T14:25:26.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:26 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/188114763' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T14:25:26.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:26 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/3510963862' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T14:25:26.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:26 smithi194 ceph-mon[45450]: pgmap v233: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-01-24T14:25:26.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:26 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/188114763' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T14:25:26.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:26 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/3510963862' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-01-24T14:25:27.486 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.484+0000 7f4e6b7fe700 1 -- 172.21.15.19:0/3035990576 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) v1 ==== 118+0+0 (secure 0 0 0) 0x7f4e64016070 con 0x7f4e74100e90 2024-01-24T14:25:27.489 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.487+0000 7f4e7c78e700 1 -- 172.21.15.19:0/3035990576 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f4e600a05a0 msgr2=0x7f4e600a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:27.489 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.487+0000 7f4e7c78e700 1 --2- 172.21.15.19:0/3035990576 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f4e600a05a0 0x7f4e600a2a60 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4e70009f50 tx=0x7f4e7000b540 comp rx=0 tx=0).stop 2024-01-24T14:25:27.489 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.488+0000 7f4e7c78e700 1 -- 172.21.15.19:0/3035990576 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4e74100e90 msgr2=0x7f4e7419b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:27.489 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.488+0000 7f4e7c78e700 1 --2- 172.21.15.19:0/3035990576 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4e74100e90 0x7f4e7419b750 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4e6400d980 tx=0x7f4e6400dc90 comp rx=0 tx=0).stop 2024-01-24T14:25:27.489 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.488+0000 7f4e7c78e700 1 -- 172.21.15.19:0/3035990576 shutdown_connections 2024-01-24T14:25:27.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.488+0000 7f4e7c78e700 1 --2- 172.21.15.19:0/3035990576 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e74101830 0x7f4e7419dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:27.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.488+0000 7f4e7c78e700 1 --2- 172.21.15.19:0/3035990576 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e740ffca0 0x7f4e74069be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:27.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.488+0000 7f4e7c78e700 1 --2- 172.21.15.19:0/3035990576 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f4e600a05a0 0x7f4e600a2a60 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:27.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.488+0000 7f4e7c78e700 1 --2- 172.21.15.19:0/3035990576 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4e74100e90 0x7f4e7419b750 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:27.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.488+0000 7f4e7c78e700 1 -- 172.21.15.19:0/3035990576 >> 172.21.15.19:0/3035990576 conn(0x7f4e740fb210 msgr2=0x7f4e740fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:27.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.488+0000 7f4e7c78e700 1 -- 172.21.15.19:0/3035990576 shutdown_connections 2024-01-24T14:25:27.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.488+0000 7f4e7c78e700 1 -- 172.21.15.19:0/3035990576 wait complete. 2024-01-24T14:25:27.505 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-01-24T14:25:27.505 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-01-24T14:25:27.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.649+0000 7f2983101700 1 -- 172.21.15.19:0/404746070 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f297c0ff750 msgr2=0x7f297c109300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:27.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.649+0000 7f2983101700 1 --2- 172.21.15.19:0/404746070 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f297c0ff750 0x7f297c109300 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f297400b0a0 tx=0x7f297400b3b0 comp rx=0 tx=0).stop 2024-01-24T14:25:27.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.649+0000 7f2983101700 1 -- 172.21.15.19:0/404746070 shutdown_connections 2024-01-24T14:25:27.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.649+0000 7f2983101700 1 --2- 172.21.15.19:0/404746070 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f297c0ff750 0x7f297c109300 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:27.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.649+0000 7f2983101700 1 --2- 172.21.15.19:0/404746070 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f297c102320 0x7f297c102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:27.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.649+0000 7f2983101700 1 --2- 172.21.15.19:0/404746070 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f297c0fed90 0x7f297c0ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:27.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.649+0000 7f2983101700 1 -- 172.21.15.19:0/404746070 >> 172.21.15.19:0/404746070 conn(0x7f297c0fa910 msgr2=0x7f297c0fcd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:27.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.649+0000 7f2983101700 1 -- 172.21.15.19:0/404746070 shutdown_connections 2024-01-24T14:25:27.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.649+0000 7f2983101700 1 -- 172.21.15.19:0/404746070 wait complete. 2024-01-24T14:25:27.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.649+0000 7f2983101700 1 Processor -- start 2024-01-24T14:25:27.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.649+0000 7f2983101700 1 -- start start 2024-01-24T14:25:27.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.650+0000 7f2983101700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f297c0fed90 0x7f297c101a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:27.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.650+0000 7f2983101700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f297c0ff750 0x7f297c100060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:27.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.650+0000 7f2983101700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f297c102320 0x7f297c1005a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:27.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.650+0000 7f2983101700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f297c100d40 con 0x7f297c0fed90 2024-01-24T14:25:27.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.650+0000 7f2983101700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f297c100eb0 con 0x7f297c102320 2024-01-24T14:25:27.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.650+0000 7f2983101700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f297c1a0b60 con 0x7f297c0ff750 2024-01-24T14:25:27.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.650+0000 7f2980e9d700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f297c0fed90 0x7f297c101a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:27.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.650+0000 7f2980e9d700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f297c0fed90 0x7f297c101a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:58854/0 (socket says 172.21.15.19:58854) 2024-01-24T14:25:27.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.650+0000 7f2980e9d700 1 -- 172.21.15.19:0/254074868 learned_addr learned my addr 172.21.15.19:0/254074868 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:25:27.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.650+0000 7f2973fff700 1 --2- 172.21.15.19:0/254074868 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f297c0ff750 0x7f297c100060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:27.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.650+0000 7f298169e700 1 --2- 172.21.15.19:0/254074868 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f297c102320 0x7f297c1005a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:27.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.650+0000 7f2980e9d700 1 -- 172.21.15.19:0/254074868 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f297c0ff750 msgr2=0x7f297c100060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:27.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.650+0000 7f2980e9d700 1 --2- 172.21.15.19:0/254074868 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f297c0ff750 0x7f297c100060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:27.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.650+0000 7f2980e9d700 1 -- 172.21.15.19:0/254074868 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f297c102320 msgr2=0x7f297c1005a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:27.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.650+0000 7f2980e9d700 1 --2- 172.21.15.19:0/254074868 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f297c102320 0x7f297c1005a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:27.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.650+0000 7f2980e9d700 1 -- 172.21.15.19:0/254074868 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2974009d00 con 0x7f297c0fed90 2024-01-24T14:25:27.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.650+0000 7f298169e700 1 --2- 172.21.15.19:0/254074868 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f297c102320 0x7f297c1005a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T14:25:27.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.650+0000 7f2980e9d700 1 --2- 172.21.15.19:0/254074868 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f297c0fed90 0x7f297c101a20 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f297800e970 tx=0x7f297800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:27.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.651+0000 7f2971ffb700 1 -- 172.21.15.19:0/254074868 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2978009790 con 0x7f297c0fed90 2024-01-24T14:25:27.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.651+0000 7f2983101700 1 -- 172.21.15.19:0/254074868 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f297c1a0e40 con 0x7f297c0fed90 2024-01-24T14:25:27.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.651+0000 7f2971ffb700 1 -- 172.21.15.19:0/254074868 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f297800eec0 con 0x7f297c0fed90 2024-01-24T14:25:27.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.651+0000 7f2971ffb700 1 -- 172.21.15.19:0/254074868 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2978003cf0 con 0x7f297c0fed90 2024-01-24T14:25:27.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.651+0000 7f2983101700 1 -- 172.21.15.19:0/254074868 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f297c1a1270 con 0x7f297c0fed90 2024-01-24T14:25:27.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.652+0000 7f2983101700 1 -- 172.21.15.19:0/254074868 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f297c107e40 con 0x7f297c0fed90 2024-01-24T14:25:27.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.653+0000 7f2971ffb700 1 -- 172.21.15.19:0/254074868 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f297800ccd0 con 0x7f297c0fed90 2024-01-24T14:25:27.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.654+0000 7f2971ffb700 1 --2- 172.21.15.19:0/254074868 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f29640a0520 0x7f29640a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:27.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.654+0000 7f2971ffb700 1 -- 172.21.15.19:0/254074868 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f2978014070 con 0x7f297c0fed90 2024-01-24T14:25:27.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.654+0000 7f2973fff700 1 --2- 172.21.15.19:0/254074868 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f29640a0520 0x7f29640a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:27.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.654+0000 7f2973fff700 1 --2- 172.21.15.19:0/254074868 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f29640a0520 0x7f29640a29e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f297c067e10 tx=0x7f296c007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:27.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.658+0000 7f2971ffb700 1 -- 172.21.15.19:0/254074868 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f2978095750 con 0x7f297c0fed90 2024-01-24T14:25:27.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:27 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/3035990576' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-01-24T14:25:27.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:27 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/3035990576' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-01-24T14:25:27.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:27 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/3035990576' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-01-24T14:25:27.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:27.918+0000 7f2983101700 1 -- 172.21.15.19:0/254074868 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) v1 -- 0x7f297c061a90 con 0x7f297c0fed90 2024-01-24T14:25:28.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.486+0000 7f2971ffb700 1 -- 172.21.15.19:0/254074868 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v18) v1 ==== 105+0+0 (secure 0 0 0) 0x7f2978098d70 con 0x7f297c0fed90 2024-01-24T14:25:28.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.490+0000 7f2983101700 1 -- 172.21.15.19:0/254074868 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f29640a0520 msgr2=0x7f29640a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:28.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.490+0000 7f2983101700 1 --2- 172.21.15.19:0/254074868 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f29640a0520 0x7f29640a29e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f297c067e10 tx=0x7f296c007400 comp rx=0 tx=0).stop 2024-01-24T14:25:28.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.490+0000 7f2983101700 1 -- 172.21.15.19:0/254074868 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f297c0fed90 msgr2=0x7f297c101a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:28.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.490+0000 7f2983101700 1 --2- 172.21.15.19:0/254074868 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f297c0fed90 0x7f297c101a20 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f297800e970 tx=0x7f297800ec80 comp rx=0 tx=0).stop 2024-01-24T14:25:28.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.490+0000 7f2983101700 1 -- 172.21.15.19:0/254074868 shutdown_connections 2024-01-24T14:25:28.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.490+0000 7f2983101700 1 --2- 172.21.15.19:0/254074868 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f29640a0520 0x7f29640a29e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:28.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.490+0000 7f2983101700 1 --2- 172.21.15.19:0/254074868 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f297c0fed90 0x7f297c101a20 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:28.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.490+0000 7f2983101700 1 --2- 172.21.15.19:0/254074868 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f297c102320 0x7f297c1005a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:28.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.490+0000 7f2983101700 1 --2- 172.21.15.19:0/254074868 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f297c0ff750 0x7f297c100060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:28.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.490+0000 7f2983101700 1 -- 172.21.15.19:0/254074868 >> 172.21.15.19:0/254074868 conn(0x7f297c0fa910 msgr2=0x7f297c0fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:28.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.490+0000 7f2983101700 1 -- 172.21.15.19:0/254074868 shutdown_connections 2024-01-24T14:25:28.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.490+0000 7f2983101700 1 -- 172.21.15.19:0/254074868 wait complete. 2024-01-24T14:25:28.509 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-01-24T14:25:28.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.652+0000 7f5d30703700 1 -- 172.21.15.19:0/3193478326 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5d281017c0 msgr2=0x7f5d28108070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:28.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.652+0000 7f5d30703700 1 --2- 172.21.15.19:0/3193478326 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5d281017c0 0x7f5d28108070 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5d2000b0d0 tx=0x7f5d2000b3e0 comp rx=0 tx=0).stop 2024-01-24T14:25:28.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.652+0000 7f5d30703700 1 -- 172.21.15.19:0/3193478326 shutdown_connections 2024-01-24T14:25:28.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.652+0000 7f5d30703700 1 --2- 172.21.15.19:0/3193478326 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5d281017c0 0x7f5d28108070 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:28.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.652+0000 7f5d30703700 1 --2- 172.21.15.19:0/3193478326 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5d280ffc30 0x7f5d28100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:28.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.652+0000 7f5d30703700 1 --2- 172.21.15.19:0/3193478326 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d28100e20 0x7f5d28101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:28.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.652+0000 7f5d30703700 1 -- 172.21.15.19:0/3193478326 >> 172.21.15.19:0/3193478326 conn(0x7f5d280fb1a0 msgr2=0x7f5d280fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:28.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.652+0000 7f5d30703700 1 -- 172.21.15.19:0/3193478326 shutdown_connections 2024-01-24T14:25:28.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.653+0000 7f5d30703700 1 -- 172.21.15.19:0/3193478326 wait complete. 2024-01-24T14:25:28.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.653+0000 7f5d30703700 1 Processor -- start 2024-01-24T14:25:28.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.653+0000 7f5d30703700 1 -- start start 2024-01-24T14:25:28.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.654+0000 7f5d30703700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d280ffc30 0x7f5d28068a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:28.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.654+0000 7f5d30703700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5d28100e20 0x7f5d2819b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:28.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.654+0000 7f5d30703700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5d281017c0 0x7f5d2819dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:28.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.654+0000 7f5d30703700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2819e660 con 0x7f5d28100e20 2024-01-24T14:25:28.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.654+0000 7f5d30703700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2819e7d0 con 0x7f5d280ffc30 2024-01-24T14:25:28.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.654+0000 7f5d30703700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2819e990 con 0x7f5d281017c0 2024-01-24T14:25:28.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.654+0000 7f5d2dc9e700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5d28100e20 0x7f5d2819b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:28.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.654+0000 7f5d2dc9e700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5d28100e20 0x7f5d2819b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:58860/0 (socket says 172.21.15.19:58860) 2024-01-24T14:25:28.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.654+0000 7f5d2dc9e700 1 -- 172.21.15.19:0/4045928206 learned_addr learned my addr 172.21.15.19:0/4045928206 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:25:28.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.654+0000 7f5d2e49f700 1 --2- 172.21.15.19:0/4045928206 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d280ffc30 0x7f5d28068a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:28.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.654+0000 7f5d2eca0700 1 --2- 172.21.15.19:0/4045928206 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5d281017c0 0x7f5d2819dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:28.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.654+0000 7f5d2dc9e700 1 -- 172.21.15.19:0/4045928206 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5d281017c0 msgr2=0x7f5d2819dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:28.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.654+0000 7f5d2dc9e700 1 --2- 172.21.15.19:0/4045928206 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5d281017c0 0x7f5d2819dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:28.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.654+0000 7f5d2dc9e700 1 -- 172.21.15.19:0/4045928206 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d280ffc30 msgr2=0x7f5d28068a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:28.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.654+0000 7f5d2dc9e700 1 --2- 172.21.15.19:0/4045928206 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d280ffc30 0x7f5d28068a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:28.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.654+0000 7f5d2dc9e700 1 -- 172.21.15.19:0/4045928206 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d20009d00 con 0x7f5d28100e20 2024-01-24T14:25:28.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.654+0000 7f5d2dc9e700 1 --2- 172.21.15.19:0/4045928206 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5d28100e20 0x7f5d2819b880 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5d1800d950 tx=0x7f5d1800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:28.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.655+0000 7f5d1f7fe700 1 -- 172.21.15.19:0/4045928206 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d18009940 con 0x7f5d28100e20 2024-01-24T14:25:28.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.655+0000 7f5d30703700 1 -- 172.21.15.19:0/4045928206 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d2819ec70 con 0x7f5d28100e20 2024-01-24T14:25:28.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.655+0000 7f5d1f7fe700 1 -- 172.21.15.19:0/4045928206 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5d18010460 con 0x7f5d28100e20 2024-01-24T14:25:28.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.655+0000 7f5d1f7fe700 1 -- 172.21.15.19:0/4045928206 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1800f620 con 0x7f5d28100e20 2024-01-24T14:25:28.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.655+0000 7f5d30703700 1 -- 172.21.15.19:0/4045928206 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d2819f180 con 0x7f5d28100e20 2024-01-24T14:25:28.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.656+0000 7f5d30703700 1 -- 172.21.15.19:0/4045928206 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d2810b450 con 0x7f5d28100e20 2024-01-24T14:25:28.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.657+0000 7f5d1f7fe700 1 -- 172.21.15.19:0/4045928206 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f5d180108d0 con 0x7f5d28100e20 2024-01-24T14:25:28.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.657+0000 7f5d1f7fe700 1 --2- 172.21.15.19:0/4045928206 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f5d140a0520 0x7f5d140a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:28.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.658+0000 7f5d1f7fe700 1 -- 172.21.15.19:0/4045928206 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f5d180cb9a0 con 0x7f5d28100e20 2024-01-24T14:25:28.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.658+0000 7f5d2e49f700 1 --2- 172.21.15.19:0/4045928206 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f5d140a0520 0x7f5d140a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:28.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.658+0000 7f5d2e49f700 1 --2- 172.21.15.19:0/4045928206 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f5d140a0520 0x7f5d140a29e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5d28100c80 tx=0x7f5d2400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:28.662 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.661+0000 7f5d1f7fe700 1 -- 172.21.15.19:0/4045928206 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f5d18095350 con 0x7f5d28100e20 2024-01-24T14:25:28.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:28 smithi019 ceph-mon[40663]: pgmap v234: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-01-24T14:25:28.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:28 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/3035990576' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-01-24T14:25:28.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:28 smithi019 ceph-mon[40663]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-01-24T14:25:28.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:28 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/254074868' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-01-24T14:25:28.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:28 smithi123 ceph-mon[44966]: pgmap v234: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-01-24T14:25:28.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:28 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/3035990576' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-01-24T14:25:28.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:28 smithi123 ceph-mon[44966]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-01-24T14:25:28.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:28 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/254074868' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-01-24T14:25:28.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:28 smithi194 ceph-mon[45450]: pgmap v234: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-01-24T14:25:28.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:28 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/3035990576' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-01-24T14:25:28.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:28 smithi194 ceph-mon[45450]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-01-24T14:25:28.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:28 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/254074868' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-01-24T14:25:28.923 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:28.922+0000 7f5d30703700 1 -- 172.21.15.19:0/4045928206 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f5d28106b30 con 0x7f5d28100e20 2024-01-24T14:25:29.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.446+0000 7f5d1f7fe700 1 -- 172.21.15.19:0/4045928206 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v20) v1 ==== 121+0+0 (secure 0 0 0) 0x7f5d180160c0 con 0x7f5d28100e20 2024-01-24T14:25:29.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.449+0000 7f5d30703700 1 -- 172.21.15.19:0/4045928206 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f5d140a0520 msgr2=0x7f5d140a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:29.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.449+0000 7f5d30703700 1 --2- 172.21.15.19:0/4045928206 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f5d140a0520 0x7f5d140a29e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5d28100c80 tx=0x7f5d2400b410 comp rx=0 tx=0).stop 2024-01-24T14:25:29.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.449+0000 7f5d30703700 1 -- 172.21.15.19:0/4045928206 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5d28100e20 msgr2=0x7f5d2819b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:29.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.449+0000 7f5d30703700 1 --2- 172.21.15.19:0/4045928206 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5d28100e20 0x7f5d2819b880 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5d1800d950 tx=0x7f5d1800dc60 comp rx=0 tx=0).stop 2024-01-24T14:25:29.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.449+0000 7f5d30703700 1 -- 172.21.15.19:0/4045928206 shutdown_connections 2024-01-24T14:25:29.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.449+0000 7f5d30703700 1 --2- 172.21.15.19:0/4045928206 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5d281017c0 0x7f5d2819dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:29.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.449+0000 7f5d30703700 1 --2- 172.21.15.19:0/4045928206 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d280ffc30 0x7f5d28068a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:29.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.449+0000 7f5d30703700 1 --2- 172.21.15.19:0/4045928206 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f5d140a0520 0x7f5d140a29e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:29.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.449+0000 7f5d30703700 1 --2- 172.21.15.19:0/4045928206 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5d28100e20 0x7f5d2819b880 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:29.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.449+0000 7f5d30703700 1 -- 172.21.15.19:0/4045928206 >> 172.21.15.19:0/4045928206 conn(0x7f5d280fb1a0 msgr2=0x7f5d280fd3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:29.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.449+0000 7f5d30703700 1 -- 172.21.15.19:0/4045928206 shutdown_connections 2024-01-24T14:25:29.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.449+0000 7f5d30703700 1 -- 172.21.15.19:0/4045928206 wait complete. 2024-01-24T14:25:29.470 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-01-24T14:25:29.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.614+0000 7f4acb4f1700 1 -- 172.21.15.19:0/1191383720 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4ac4100e60 msgr2=0x7f4ac41012c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:29.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.614+0000 7f4acb4f1700 1 --2- 172.21.15.19:0/1191383720 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4ac4100e60 0x7f4ac41012c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4ab40099c0 tx=0x7f4ab4009cd0 comp rx=0 tx=0).stop 2024-01-24T14:25:29.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.614+0000 7f4acb4f1700 1 -- 172.21.15.19:0/1191383720 shutdown_connections 2024-01-24T14:25:29.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.614+0000 7f4acb4f1700 1 --2- 172.21.15.19:0/1191383720 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ac4101800 0x7f4ac41080b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:29.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.614+0000 7f4acb4f1700 1 --2- 172.21.15.19:0/1191383720 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4ac40ffc70 0x7f4ac4100090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:29.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.614+0000 7f4acb4f1700 1 --2- 172.21.15.19:0/1191383720 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4ac4100e60 0x7f4ac41012c0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:29.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.614+0000 7f4acb4f1700 1 -- 172.21.15.19:0/1191383720 >> 172.21.15.19:0/1191383720 conn(0x7f4ac40fb200 msgr2=0x7f4ac40fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:29.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.614+0000 7f4acb4f1700 1 -- 172.21.15.19:0/1191383720 shutdown_connections 2024-01-24T14:25:29.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.614+0000 7f4acb4f1700 1 -- 172.21.15.19:0/1191383720 wait complete. 2024-01-24T14:25:29.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.615+0000 7f4acb4f1700 1 Processor -- start 2024-01-24T14:25:29.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.615+0000 7f4acb4f1700 1 -- start start 2024-01-24T14:25:29.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.616+0000 7f4acb4f1700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4ac40ffc70 0x7f4ac40699e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:29.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.616+0000 7f4acb4f1700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ac4100e60 0x7f4ac419b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:29.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.616+0000 7f4acb4f1700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4ac4101800 0x7f4ac419ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:29.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.616+0000 7f4acb4f1700 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac419e5c0 con 0x7f4ac4101800 2024-01-24T14:25:29.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.616+0000 7f4acb4f1700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac419e730 con 0x7f4ac4100e60 2024-01-24T14:25:29.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.616+0000 7f4acb4f1700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac419e8f0 con 0x7f4ac40ffc70 2024-01-24T14:25:29.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.616+0000 7f4ac9a8e700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4ac4101800 0x7f4ac419ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:29.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.616+0000 7f4ac9a8e700 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4ac4101800 0x7f4ac419ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:58878/0 (socket says 172.21.15.19:58878) 2024-01-24T14:25:29.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.616+0000 7f4ac9a8e700 1 -- 172.21.15.19:0/2833132224 learned_addr learned my addr 172.21.15.19:0/2833132224 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-01-24T14:25:29.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.616+0000 7f4ac8a8c700 1 --2- 172.21.15.19:0/2833132224 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ac4100e60 0x7f4ac419b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:29.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.616+0000 7f4ac928d700 1 --2- 172.21.15.19:0/2833132224 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4ac40ffc70 0x7f4ac40699e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:29.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.616+0000 7f4ac9a8e700 1 -- 172.21.15.19:0/2833132224 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4ac40ffc70 msgr2=0x7f4ac40699e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:29.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.616+0000 7f4ac9a8e700 1 --2- 172.21.15.19:0/2833132224 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4ac40ffc70 0x7f4ac40699e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:29.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.616+0000 7f4ac9a8e700 1 -- 172.21.15.19:0/2833132224 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ac4100e60 msgr2=0x7f4ac419b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:29.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.616+0000 7f4ac9a8e700 1 --2- 172.21.15.19:0/2833132224 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ac4100e60 0x7f4ac419b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:29.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.616+0000 7f4ac9a8e700 1 -- 172.21.15.19:0/2833132224 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ab40095f0 con 0x7f4ac4101800 2024-01-24T14:25:29.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.616+0000 7f4ac928d700 1 --2- 172.21.15.19:0/2833132224 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4ac40ffc70 0x7f4ac40699e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-24T14:25:29.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.616+0000 7f4ac9a8e700 1 --2- 172.21.15.19:0/2833132224 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4ac4101800 0x7f4ac419ddf0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4abc00ef30 tx=0x7f4abc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:29.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.616+0000 7f4aba7fc700 1 -- 172.21.15.19:0/2833132224 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4abc018070 con 0x7f4ac4101800 2024-01-24T14:25:29.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.617+0000 7f4acb4f1700 1 -- 172.21.15.19:0/2833132224 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ac419eb70 con 0x7f4ac4101800 2024-01-24T14:25:29.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.617+0000 7f4aba7fc700 1 -- 172.21.15.19:0/2833132224 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4abc00f040 con 0x7f4ac4101800 2024-01-24T14:25:29.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.617+0000 7f4aba7fc700 1 -- 172.21.15.19:0/2833132224 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4abc00cdb0 con 0x7f4ac4101800 2024-01-24T14:25:29.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.617+0000 7f4acb4f1700 1 -- 172.21.15.19:0/2833132224 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ac419f000 con 0x7f4ac4101800 2024-01-24T14:25:29.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.618+0000 7f4acb4f1700 1 -- 172.21.15.19:0/2833132224 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ac4107200 con 0x7f4ac4101800 2024-01-24T14:25:29.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.619+0000 7f4aba7fc700 1 -- 172.21.15.19:0/2833132224 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 16) v1 ==== 135070+0+0 (secure 0 0 0) 0x7f4abc003a00 con 0x7f4ac4101800 2024-01-24T14:25:29.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.619+0000 7f4aba7fc700 1 --2- 172.21.15.19:0/2833132224 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f4ab00a0520 0x7f4ab00a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-24T14:25:29.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.619+0000 7f4aba7fc700 1 -- 172.21.15.19:0/2833132224 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f4abc0cc0b0 con 0x7f4ac4101800 2024-01-24T14:25:29.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.619+0000 7f4ac928d700 1 --2- 172.21.15.19:0/2833132224 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f4ab00a0520 0x7f4ab00a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-24T14:25:29.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.620+0000 7f4ac928d700 1 --2- 172.21.15.19:0/2833132224 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f4ab00a0520 0x7f4ab00a29e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4ac4100cc0 tx=0x7f4ac0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-24T14:25:29.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.623+0000 7f4aba7fc700 1 -- 172.21.15.19:0/2833132224 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180484 (secure 0 0 0) 0x7f4abc095af0 con 0x7f4ac4101800 2024-01-24T14:25:29.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:29 smithi019 ceph-mon[40663]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-01-24T14:25:29.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:29 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/254074868' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-01-24T14:25:29.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:29 smithi019 ceph-mon[40663]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-01-24T14:25:29.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:29 smithi019 ceph-mon[40663]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-01-24T14:25:29.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:29 smithi019 ceph-mon[40663]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-01-24T14:25:29.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:29 smithi019 ceph-mon[40663]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-01-24T14:25:29.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:29 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/4045928206' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-01-24T14:25:29.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:29 smithi019 ceph-mon[40663]: pgmap v235: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-01-24T14:25:29.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:29 smithi019 ceph-mon[40663]: mds.? [v2:172.21.15.194:6838/810249223,v1:172.21.15.194:6839/810249223] up:active 2024-01-24T14:25:29.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:29 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/4045928206' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-01-24T14:25:29.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:29 smithi019 ceph-mon[40663]: daemon mds.f assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-01-24T14:25:29.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:29 smithi019 ceph-mon[40663]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 2 up:standby-replay 8 up:standby 2024-01-24T14:25:29.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:29 smithi019 ceph-mon[40663]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:creating} 2 up:standby-replay 7 up:standby 2024-01-24T14:25:29.737 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:29 smithi019 ceph-mon[40663]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-01-24T14:25:29.799 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:29 smithi123 ceph-mon[44966]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-01-24T14:25:29.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:29 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/254074868' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-01-24T14:25:29.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:29 smithi123 ceph-mon[44966]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-01-24T14:25:29.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:29 smithi123 ceph-mon[44966]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-01-24T14:25:29.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:29 smithi123 ceph-mon[44966]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-01-24T14:25:29.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:29 smithi123 ceph-mon[44966]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-01-24T14:25:29.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:29 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/4045928206' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-01-24T14:25:29.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:29 smithi123 ceph-mon[44966]: pgmap v235: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-01-24T14:25:29.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:29 smithi123 ceph-mon[44966]: mds.? [v2:172.21.15.194:6838/810249223,v1:172.21.15.194:6839/810249223] up:active 2024-01-24T14:25:29.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:29 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/4045928206' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-01-24T14:25:29.801 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:29 smithi123 ceph-mon[44966]: daemon mds.f assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-01-24T14:25:29.801 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:29 smithi123 ceph-mon[44966]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 2 up:standby-replay 8 up:standby 2024-01-24T14:25:29.801 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:29 smithi123 ceph-mon[44966]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:creating} 2 up:standby-replay 7 up:standby 2024-01-24T14:25:29.801 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:29 smithi123 ceph-mon[44966]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-01-24T14:25:29.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:29 smithi194 ceph-mon[45450]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-01-24T14:25:29.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:29 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/254074868' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-01-24T14:25:29.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:29 smithi194 ceph-mon[45450]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-01-24T14:25:29.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:29 smithi194 ceph-mon[45450]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-01-24T14:25:29.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:29 smithi194 ceph-mon[45450]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-01-24T14:25:29.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:29 smithi194 ceph-mon[45450]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-01-24T14:25:29.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:29 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/4045928206' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-01-24T14:25:29.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:29 smithi194 ceph-mon[45450]: pgmap v235: 129 pgs: 129 active+clean; 581 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-01-24T14:25:29.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:29 smithi194 ceph-mon[45450]: mds.? [v2:172.21.15.194:6838/810249223,v1:172.21.15.194:6839/810249223] up:active 2024-01-24T14:25:29.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:29 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/4045928206' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-01-24T14:25:29.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:29 smithi194 ceph-mon[45450]: daemon mds.f assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-01-24T14:25:29.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:29 smithi194 ceph-mon[45450]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 2 up:standby-replay 8 up:standby 2024-01-24T14:25:29.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:29 smithi194 ceph-mon[45450]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:creating} 2 up:standby-replay 7 up:standby 2024-01-24T14:25:29.845 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:29 smithi194 ceph-mon[45450]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-01-24T14:25:29.885 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:29.884+0000 7f4acb4f1700 1 -- 172.21.15.19:0/2833132224 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f4ac402cc70 con 0x7f4ac4101800 2024-01-24T14:25:30.454 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:30.453+0000 7f4aba7fc700 1 -- 172.21.15.19:0/2833132224 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v22) v1 ==== 115+0+0 (secure 0 0 0) 0x7f4abc099110 con 0x7f4ac4101800 2024-01-24T14:25:30.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:30.456+0000 7f4acb4f1700 1 -- 172.21.15.19:0/2833132224 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f4ab00a0520 msgr2=0x7f4ab00a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:30.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:30.456+0000 7f4acb4f1700 1 --2- 172.21.15.19:0/2833132224 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f4ab00a0520 0x7f4ab00a29e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4ac4100cc0 tx=0x7f4ac0009260 comp rx=0 tx=0).stop 2024-01-24T14:25:30.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:30.456+0000 7f4acb4f1700 1 -- 172.21.15.19:0/2833132224 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4ac4101800 msgr2=0x7f4ac419ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-24T14:25:30.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:30.456+0000 7f4acb4f1700 1 --2- 172.21.15.19:0/2833132224 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4ac4101800 0x7f4ac419ddf0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4abc00ef30 tx=0x7f4abc00c3a0 comp rx=0 tx=0).stop 2024-01-24T14:25:30.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:30.456+0000 7f4acb4f1700 1 -- 172.21.15.19:0/2833132224 shutdown_connections 2024-01-24T14:25:30.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:30.456+0000 7f4acb4f1700 1 --2- 172.21.15.19:0/2833132224 >> [v2:172.21.15.19:6800/1634350318,v1:172.21.15.19:6801/1634350318] conn(0x7f4ab00a0520 0x7f4ab00a29e0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:30.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:30.456+0000 7f4acb4f1700 1 --2- 172.21.15.19:0/2833132224 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4ac4101800 0x7f4ac419ddf0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:30.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:30.456+0000 7f4acb4f1700 1 --2- 172.21.15.19:0/2833132224 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4ac40ffc70 0x7f4ac40699e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:30.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:30.456+0000 7f4acb4f1700 1 --2- 172.21.15.19:0/2833132224 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ac4100e60 0x7f4ac419b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-24T14:25:30.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:30.456+0000 7f4acb4f1700 1 -- 172.21.15.19:0/2833132224 >> 172.21.15.19:0/2833132224 conn(0x7f4ac40fb200 msgr2=0x7f4ac4104340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-24T14:25:30.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:30.456+0000 7f4acb4f1700 1 -- 172.21.15.19:0/2833132224 shutdown_connections 2024-01-24T14:25:30.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-01-24T14:25:30.456+0000 7f4acb4f1700 1 -- 172.21.15.19:0/2833132224 wait complete. 2024-01-24T14:25:30.476 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated'} subvols for filesystem 'cephfs' 2024-01-24T14:25:30.476 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_4e3182a65800aed6d4aa2f6b86da3cfd72188134/qa/tasks/cephadm.py", line 1051, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/git.ceph.com_ceph-c_4e3182a65800aed6d4aa2f6b86da3cfd72188134/qa/tasks/cephfs/filesystem.py", line 532, in __init__ self.create() File "/home/teuthworker/src/git.ceph.com_ceph-c_4e3182a65800aed6d4aa2f6b86da3cfd72188134/qa/tasks/cephfs/filesystem.py", line 762, in create self.run_ceph_cmd('fs', 'subvolumegroup', 'create', self.name, 'qa') AttributeError: 'Filesystem' object has no attribute 'run_ceph_cmd' 2024-01-24T14:25:30.479 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-01-24T14:25:30.512 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-01-24T14:25:30.549 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-01-24T14:25:30.587 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-01-24T14:25:30.588 DEBUG:teuthology.orchestra.run.smithi019:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-01-24T14:25:30.610 DEBUG:teuthology.orchestra.run.smithi123:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-01-24T14:25:30.629 DEBUG:teuthology.orchestra.run.smithi194:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-01-24T14:25:30.648 INFO:tasks.cephadm:Stopping all daemons... 2024-01-24T14:25:30.649 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-01-24T14:25:30.649 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl stop ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mon.a 2024-01-24T14:25:30.735 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:30 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/2833132224' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-01-24T14:25:30.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:30 smithi019 ceph-mon[40663]: mds.? [v2:172.21.15.194:6834/456696823,v1:172.21.15.194:6835/456696823] up:active 2024-01-24T14:25:30.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:30 smithi019 ceph-mon[40663]: from='client.? 172.21.15.19:0/2833132224' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-01-24T14:25:30.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:30 smithi019 ceph-mon[40663]: daemon mds.e assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-01-24T14:25:30.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:30 smithi019 ceph-mon[40663]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2024-01-24T14:25:30.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:30 smithi019 ceph-mon[40663]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=f=up:active,3=e=up:creating} 2 up:standby-replay 6 up:standby 2024-01-24T14:25:30.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:30 smithi019 ceph-mon[40663]: daemon mds.e is now active in filesystem cephfs as rank 3 2024-01-24T14:25:30.736 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:30 smithi019 systemd[1]: Stopping Ceph mon.a for 654b0c8e-bac3-11ee-95b1-87774f69a715... 2024-01-24T14:25:30.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:30 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/2833132224' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-01-24T14:25:30.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:30 smithi123 ceph-mon[44966]: mds.? [v2:172.21.15.194:6834/456696823,v1:172.21.15.194:6835/456696823] up:active 2024-01-24T14:25:30.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:30 smithi123 ceph-mon[44966]: from='client.? 172.21.15.19:0/2833132224' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-01-24T14:25:30.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:30 smithi123 ceph-mon[44966]: daemon mds.e assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-01-24T14:25:30.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:30 smithi123 ceph-mon[44966]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2024-01-24T14:25:30.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:30 smithi123 ceph-mon[44966]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=f=up:active,3=e=up:creating} 2 up:standby-replay 6 up:standby 2024-01-24T14:25:30.800 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:30 smithi123 ceph-mon[44966]: daemon mds.e is now active in filesystem cephfs as rank 3 2024-01-24T14:25:30.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:30 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/2833132224' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-01-24T14:25:30.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:30 smithi194 ceph-mon[45450]: mds.? [v2:172.21.15.194:6834/456696823,v1:172.21.15.194:6835/456696823] up:active 2024-01-24T14:25:30.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:30 smithi194 ceph-mon[45450]: from='client.? 172.21.15.19:0/2833132224' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-01-24T14:25:30.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:30 smithi194 ceph-mon[45450]: daemon mds.e assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-01-24T14:25:30.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:30 smithi194 ceph-mon[45450]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2024-01-24T14:25:30.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:30 smithi194 ceph-mon[45450]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=f=up:active,3=e=up:creating} 2 up:standby-replay 6 up:standby 2024-01-24T14:25:30.844 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:30 smithi194 ceph-mon[45450]: daemon mds.e is now active in filesystem cephfs as rank 3 2024-01-24T14:25:31.235 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:30 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mon-a[40640]: 2024-01-24T14:25:30.909+0000 7fadac160700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T14:25:31.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:30 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mon-a[40640]: 2024-01-24T14:25:30.909+0000 7fadac160700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-01-24T14:25:31.236 INFO:journalctl@ceph.mon.a.smithi019.stdout:Jan 24 14:25:31 smithi019 bash[97074]: ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mon-a 2024-01-24T14:25:31.477 DEBUG:teuthology.orchestra.run.smithi019:> sudo pkill -f 'journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mon.a.service' 2024-01-24T14:25:31.534 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T14:25:31.534 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-01-24T14:25:31.534 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-01-24T14:25:31.534 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mon.b 2024-01-24T14:25:31.884 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:31 smithi123 systemd[1]: Stopping Ceph mon.b for 654b0c8e-bac3-11ee-95b1-87774f69a715... 2024-01-24T14:25:32.178 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:31 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mon-b[44943]: 2024-01-24T14:25:31.881+0000 7fb422666700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T14:25:32.178 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:31 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mon-b[44943]: 2024-01-24T14:25:31.881+0000 7fb422666700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-01-24T14:25:32.461 INFO:journalctl@ceph.mon.b.smithi123.stdout:Jan 24 14:25:32 smithi123 bash[82667]: ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mon-b 2024-01-24T14:25:32.554 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mon.b.service' 2024-01-24T14:25:32.608 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T14:25:32.608 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-01-24T14:25:32.608 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-01-24T14:25:32.608 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mon.c 2024-01-24T14:25:32.799 INFO:journalctl@ceph.mds.b.smithi123.stdout:Jan 24 14:25:32 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-b[81360]: 2024-01-24T14:25:32.458+0000 7f37f6c96700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-01-24T14:25:32.800 INFO:journalctl@ceph.mds.b.smithi123.stdout:Jan 24 14:25:32 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-b[81360]: 2024-01-24T14:25:32.458+0000 7f37f6c96700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-01-24T14:25:33.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:32 smithi194 systemd[1]: Stopping Ceph mon.c for 654b0c8e-bac3-11ee-95b1-87774f69a715... 2024-01-24T14:25:33.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:32 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mon-c[45427]: 2024-01-24T14:25:32.849+0000 7fc8dbe0a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T14:25:33.094 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:32 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mon-c[45427]: 2024-01-24T14:25:32.849+0000 7fc8dbe0a700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-01-24T14:25:33.577 INFO:journalctl@ceph.mon.c.smithi194.stdout:Jan 24 14:25:33 smithi194 bash[84468]: ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mon-c 2024-01-24T14:25:33.585 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mon.c.service' 2024-01-24T14:25:33.692 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T14:25:33.692 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-01-24T14:25:33.692 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-01-24T14:25:33.692 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl stop ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mgr.x 2024-01-24T14:25:33.978 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:25:33 smithi019 systemd[1]: Stopping Ceph mgr.x for 654b0c8e-bac3-11ee-95b1-87774f69a715... 2024-01-24T14:25:33.978 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:25:33 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x[40914]: 2024-01-24T14:25:33.950+0000 7f99d13fc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T14:25:34.235 INFO:journalctl@ceph.mgr.x.smithi019.stdout:Jan 24 14:25:34 smithi019 bash[97569]: ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-x 2024-01-24T14:25:34.458 DEBUG:teuthology.orchestra.run.smithi019:> sudo pkill -f 'journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mgr.x.service' 2024-01-24T14:25:34.502 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T14:25:34.502 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-01-24T14:25:34.502 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-01-24T14:25:34.503 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mgr.y 2024-01-24T14:25:34.799 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Jan 24 14:25:34 smithi123 systemd[1]: Stopping Ceph mgr.y for 654b0c8e-bac3-11ee-95b1-87774f69a715... 2024-01-24T14:25:35.274 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Jan 24 14:25:34 smithi123 bash[82891]: ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-y 2024-01-24T14:25:35.275 INFO:journalctl@ceph.mgr.y.smithi123.stdout:Jan 24 14:25:35 smithi123 systemd[1]: ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-01-24T14:25:35.281 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mgr.y.service' 2024-01-24T14:25:35.318 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T14:25:35.318 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-01-24T14:25:35.318 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-01-24T14:25:35.318 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mgr.z 2024-01-24T14:25:35.672 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jan 24 14:25:35 smithi194 systemd[1]: Stopping Ceph mgr.z for 654b0c8e-bac3-11ee-95b1-87774f69a715... 2024-01-24T14:25:36.060 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jan 24 14:25:35 smithi194 bash[84626]: ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mgr-z 2024-01-24T14:25:36.060 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Jan 24 14:25:35 smithi194 systemd[1]: ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mgr.z.service: Main process exited, code=exited, status=143/n/a 2024-01-24T14:25:36.067 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mgr.z.service' 2024-01-24T14:25:36.107 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T14:25:36.108 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-01-24T14:25:36.108 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-01-24T14:25:36.108 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl stop ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@osd.0 2024-01-24T14:25:36.485 INFO:journalctl@ceph.osd.0.smithi019.stdout:Jan 24 14:25:36 smithi019 systemd[1]: Stopping Ceph osd.0 for 654b0c8e-bac3-11ee-95b1-87774f69a715... 2024-01-24T14:25:36.486 INFO:journalctl@ceph.osd.0.smithi019.stdout:Jan 24 14:25:36 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-0[59108]: 2024-01-24T14:25:36.374+0000 7fe1d9258700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T14:25:36.486 INFO:journalctl@ceph.osd.0.smithi019.stdout:Jan 24 14:25:36 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-0[59108]: 2024-01-24T14:25:36.374+0000 7fe1d9258700 -1 osd.0 82 *** Got signal Terminated *** 2024-01-24T14:25:36.486 INFO:journalctl@ceph.osd.0.smithi019.stdout:Jan 24 14:25:36 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-0[59108]: 2024-01-24T14:25:36.374+0000 7fe1d9258700 -1 osd.0 82 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-01-24T14:25:42.236 INFO:journalctl@ceph.osd.0.smithi019.stdout:Jan 24 14:25:41 smithi019 bash[97712]: ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-0 2024-01-24T14:25:43.141 DEBUG:teuthology.orchestra.run.smithi019:> sudo pkill -f 'journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@osd.0.service' 2024-01-24T14:25:43.193 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T14:25:43.194 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-01-24T14:25:43.194 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-01-24T14:25:43.194 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@osd.1 2024-01-24T14:25:43.549 INFO:journalctl@ceph.osd.1.smithi123.stdout:Jan 24 14:25:43 smithi123 systemd[1]: Stopping Ceph osd.1 for 654b0c8e-bac3-11ee-95b1-87774f69a715... 2024-01-24T14:25:43.549 INFO:journalctl@ceph.osd.1.smithi123.stdout:Jan 24 14:25:43 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-1[49931]: 2024-01-24T14:25:43.439+0000 7fb6915fb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T14:25:43.549 INFO:journalctl@ceph.osd.1.smithi123.stdout:Jan 24 14:25:43 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-1[49931]: 2024-01-24T14:25:43.439+0000 7fb6915fb700 -1 osd.1 82 *** Got signal Terminated *** 2024-01-24T14:25:43.550 INFO:journalctl@ceph.osd.1.smithi123.stdout:Jan 24 14:25:43 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-1[49931]: 2024-01-24T14:25:43.439+0000 7fb6915fb700 -1 osd.1 82 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-01-24T14:25:49.299 INFO:journalctl@ceph.osd.1.smithi123.stdout:Jan 24 14:25:48 smithi123 bash[83030]: ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-1 2024-01-24T14:25:50.210 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@osd.1.service' 2024-01-24T14:25:50.250 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T14:25:50.250 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-01-24T14:25:50.251 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-01-24T14:25:50.251 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@osd.2 2024-01-24T14:25:50.594 INFO:journalctl@ceph.osd.2.smithi194.stdout:Jan 24 14:25:50 smithi194 systemd[1]: Stopping Ceph osd.2 for 654b0c8e-bac3-11ee-95b1-87774f69a715... 2024-01-24T14:25:50.594 INFO:journalctl@ceph.osd.2.smithi194.stdout:Jan 24 14:25:50 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-2[51247]: 2024-01-24T14:25:50.505+0000 7f505143a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T14:25:50.594 INFO:journalctl@ceph.osd.2.smithi194.stdout:Jan 24 14:25:50 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-2[51247]: 2024-01-24T14:25:50.505+0000 7f505143a700 -1 osd.2 82 *** Got signal Terminated *** 2024-01-24T14:25:50.595 INFO:journalctl@ceph.osd.2.smithi194.stdout:Jan 24 14:25:50 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-2[51247]: 2024-01-24T14:25:50.505+0000 7f505143a700 -1 osd.2 82 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-01-24T14:25:56.594 INFO:journalctl@ceph.osd.2.smithi194.stdout:Jan 24 14:25:56 smithi194 bash[84769]: ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-2 2024-01-24T14:25:57.369 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@osd.2.service' 2024-01-24T14:25:57.410 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T14:25:57.411 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-01-24T14:25:57.411 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-01-24T14:25:57.411 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl stop ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@osd.3 2024-01-24T14:25:57.736 INFO:journalctl@ceph.osd.3.smithi019.stdout:Jan 24 14:25:57 smithi019 systemd[1]: Stopping Ceph osd.3 for 654b0c8e-bac3-11ee-95b1-87774f69a715... 2024-01-24T14:25:57.736 INFO:journalctl@ceph.osd.3.smithi019.stdout:Jan 24 14:25:57 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-3[67628]: 2024-01-24T14:25:57.671+0000 7fb0c2073700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T14:25:57.736 INFO:journalctl@ceph.osd.3.smithi019.stdout:Jan 24 14:25:57 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-3[67628]: 2024-01-24T14:25:57.671+0000 7fb0c2073700 -1 osd.3 82 *** Got signal Terminated *** 2024-01-24T14:25:57.736 INFO:journalctl@ceph.osd.3.smithi019.stdout:Jan 24 14:25:57 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-3[67628]: 2024-01-24T14:25:57.671+0000 7fb0c2073700 -1 osd.3 82 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-01-24T14:26:03.094 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jan 24 14:26:02 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-5[59863]: 2024-01-24T14:26:02.746+0000 7f9971fc7700 -1 osd.5 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:36.584653+0000 front 2024-01-24T14:25:36.584717+0000 (oldest deadline 2024-01-24T14:26:02.484505+0000) 2024-01-24T14:26:03.094 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:02 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:02.675+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:03.486 INFO:journalctl@ceph.osd.3.smithi019.stdout:Jan 24 14:26:03 smithi019 bash[97972]: ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-3 2024-01-24T14:26:03.594 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:03 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:03.140+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:39.060282+0000 front 2024-01-24T14:25:39.060365+0000 (oldest deadline 2024-01-24T14:26:02.560180+0000) 2024-01-24T14:26:04.094 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jan 24 14:26:03 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-5[59863]: 2024-01-24T14:26:03.788+0000 7f9971fc7700 -1 osd.5 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:36.584653+0000 front 2024-01-24T14:25:36.584717+0000 (oldest deadline 2024-01-24T14:26:02.484505+0000) 2024-01-24T14:26:04.094 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:03 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:03.721+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:04.367 DEBUG:teuthology.orchestra.run.smithi019:> sudo pkill -f 'journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@osd.3.service' 2024-01-24T14:26:04.421 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T14:26:04.421 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-01-24T14:26:04.421 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-01-24T14:26:04.421 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@osd.4 2024-01-24T14:26:04.594 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:04 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:04.166+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:39.060282+0000 front 2024-01-24T14:25:39.060365+0000 (oldest deadline 2024-01-24T14:26:02.560180+0000) 2024-01-24T14:26:04.735 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:26:04 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6[76344]: 2024-01-24T14:26:04.396+0000 7f6f362bd700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.007528+0000 front 2024-01-24T14:25:40.007600+0000 (oldest deadline 2024-01-24T14:26:04.107485+0000) 2024-01-24T14:26:04.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:04 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:04.440+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.092357+0000 front 2024-01-24T14:25:38.092298+0000 (oldest deadline 2024-01-24T14:26:03.992419+0000) 2024-01-24T14:26:04.799 INFO:journalctl@ceph.osd.4.smithi123.stdout:Jan 24 14:26:04 smithi123 systemd[1]: Stopping Ceph osd.4 for 654b0c8e-bac3-11ee-95b1-87774f69a715... 2024-01-24T14:26:04.799 INFO:journalctl@ceph.osd.4.smithi123.stdout:Jan 24 14:26:04 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-4[58441]: 2024-01-24T14:26:04.677+0000 7f0149308700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T14:26:04.799 INFO:journalctl@ceph.osd.4.smithi123.stdout:Jan 24 14:26:04 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-4[58441]: 2024-01-24T14:26:04.677+0000 7f0149308700 -1 osd.4 82 *** Got signal Terminated *** 2024-01-24T14:26:04.799 INFO:journalctl@ceph.osd.4.smithi123.stdout:Jan 24 14:26:04 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-4[58441]: 2024-01-24T14:26:04.677+0000 7f0149308700 -1 osd.4 82 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-01-24T14:26:05.094 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jan 24 14:26:04 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-5[59863]: 2024-01-24T14:26:04.786+0000 7f9971fc7700 -1 osd.5 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:36.584653+0000 front 2024-01-24T14:25:36.584717+0000 (oldest deadline 2024-01-24T14:26:02.484505+0000) 2024-01-24T14:26:05.094 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:04 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:04.753+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:05.594 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:05 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:05.206+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:39.060282+0000 front 2024-01-24T14:25:39.060365+0000 (oldest deadline 2024-01-24T14:26:02.560180+0000) 2024-01-24T14:26:05.735 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:26:05 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6[76344]: 2024-01-24T14:26:05.421+0000 7f6f362bd700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.007528+0000 front 2024-01-24T14:25:40.007600+0000 (oldest deadline 2024-01-24T14:26:04.107485+0000) 2024-01-24T14:26:05.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:05 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:05.483+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.092357+0000 front 2024-01-24T14:25:38.092298+0000 (oldest deadline 2024-01-24T14:26:03.992419+0000) 2024-01-24T14:26:06.094 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jan 24 14:26:05 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-5[59863]: 2024-01-24T14:26:05.825+0000 7f9971fc7700 -1 osd.5 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:36.584653+0000 front 2024-01-24T14:25:36.584717+0000 (oldest deadline 2024-01-24T14:26:02.484505+0000) 2024-01-24T14:26:06.094 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:05 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:05.728+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:06.594 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:06 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:06.209+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:39.060282+0000 front 2024-01-24T14:25:39.060365+0000 (oldest deadline 2024-01-24T14:26:02.560180+0000) 2024-01-24T14:26:06.735 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:26:06 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6[76344]: 2024-01-24T14:26:06.417+0000 7f6f362bd700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.007528+0000 front 2024-01-24T14:25:40.007600+0000 (oldest deadline 2024-01-24T14:26:04.107485+0000) 2024-01-24T14:26:06.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:06 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:06.473+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.092357+0000 front 2024-01-24T14:25:38.092298+0000 (oldest deadline 2024-01-24T14:26:03.992419+0000) 2024-01-24T14:26:06.799 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:06 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:06.330+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.647827+0000 front 2024-01-24T14:25:40.647938+0000 (oldest deadline 2024-01-24T14:26:05.347799+0000) 2024-01-24T14:26:07.094 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jan 24 14:26:06 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-5[59863]: 2024-01-24T14:26:06.832+0000 7f9971fc7700 -1 osd.5 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:36.584653+0000 front 2024-01-24T14:25:36.584717+0000 (oldest deadline 2024-01-24T14:26:02.484505+0000) 2024-01-24T14:26:07.094 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:06 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:06.718+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:07.594 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:07 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:07.238+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:39.060282+0000 front 2024-01-24T14:25:39.060365+0000 (oldest deadline 2024-01-24T14:26:02.560180+0000) 2024-01-24T14:26:07.736 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:26:07 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6[76344]: 2024-01-24T14:26:07.399+0000 7f6f362bd700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.007528+0000 front 2024-01-24T14:25:40.007600+0000 (oldest deadline 2024-01-24T14:26:04.107485+0000) 2024-01-24T14:26:07.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:07 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:07.451+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.092357+0000 front 2024-01-24T14:25:38.092298+0000 (oldest deadline 2024-01-24T14:26:03.992419+0000) 2024-01-24T14:26:07.799 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:07 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:07.380+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.647827+0000 front 2024-01-24T14:25:40.647938+0000 (oldest deadline 2024-01-24T14:26:05.347799+0000) 2024-01-24T14:26:08.094 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jan 24 14:26:07 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-5[59863]: 2024-01-24T14:26:07.849+0000 7f9971fc7700 -1 osd.5 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:36.584653+0000 front 2024-01-24T14:25:36.584717+0000 (oldest deadline 2024-01-24T14:26:02.484505+0000) 2024-01-24T14:26:08.094 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:07 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:07.669+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:08.594 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:08 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:08.244+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:39.060282+0000 front 2024-01-24T14:25:39.060365+0000 (oldest deadline 2024-01-24T14:26:02.560180+0000) 2024-01-24T14:26:08.735 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:08 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:08.411+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.092357+0000 front 2024-01-24T14:25:38.092298+0000 (oldest deadline 2024-01-24T14:26:03.992419+0000) 2024-01-24T14:26:08.736 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:26:08 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6[76344]: 2024-01-24T14:26:08.355+0000 7f6f362bd700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.007528+0000 front 2024-01-24T14:25:40.007600+0000 (oldest deadline 2024-01-24T14:26:04.107485+0000) 2024-01-24T14:26:08.799 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:08 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:08.330+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.647827+0000 front 2024-01-24T14:25:40.647938+0000 (oldest deadline 2024-01-24T14:26:05.347799+0000) 2024-01-24T14:26:09.094 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jan 24 14:26:08 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-5[59863]: 2024-01-24T14:26:08.842+0000 7f9971fc7700 -1 osd.5 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:36.584653+0000 front 2024-01-24T14:25:36.584717+0000 (oldest deadline 2024-01-24T14:26:02.484505+0000) 2024-01-24T14:26:09.094 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:08 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:08.662+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:09.594 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:09 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:09.250+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:39.060282+0000 front 2024-01-24T14:25:39.060365+0000 (oldest deadline 2024-01-24T14:26:02.560180+0000) 2024-01-24T14:26:09.736 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:26:09 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6[76344]: 2024-01-24T14:26:09.360+0000 7f6f362bd700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.007528+0000 front 2024-01-24T14:25:40.007600+0000 (oldest deadline 2024-01-24T14:26:04.107485+0000) 2024-01-24T14:26:09.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:09 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:09.423+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.092357+0000 front 2024-01-24T14:25:38.092298+0000 (oldest deadline 2024-01-24T14:26:03.992419+0000) 2024-01-24T14:26:09.799 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:09 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:09.377+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.647827+0000 front 2024-01-24T14:25:40.647938+0000 (oldest deadline 2024-01-24T14:26:05.347799+0000) 2024-01-24T14:26:09.799 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:09 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:09.377+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:45.348138+0000 front 2024-01-24T14:25:45.348156+0000 (oldest deadline 2024-01-24T14:26:08.848352+0000) 2024-01-24T14:26:10.094 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jan 24 14:26:09 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-5[59863]: 2024-01-24T14:26:09.867+0000 7f9971fc7700 -1 osd.5 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:36.584653+0000 front 2024-01-24T14:25:36.584717+0000 (oldest deadline 2024-01-24T14:26:02.484505+0000) 2024-01-24T14:26:10.094 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:09 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:09.707+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:10.371 INFO:journalctl@ceph.osd.4.smithi123.stdout:Jan 24 14:26:10 smithi123 bash[83300]: ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-4 2024-01-24T14:26:10.594 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:10 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:10.222+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:39.060282+0000 front 2024-01-24T14:25:39.060365+0000 (oldest deadline 2024-01-24T14:26:02.560180+0000) 2024-01-24T14:26:10.594 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:10 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:10.222+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:47.261420+0000 front 2024-01-24T14:25:47.261437+0000 (oldest deadline 2024-01-24T14:26:10.161656+0000) 2024-01-24T14:26:10.706 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:10 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:10.370+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.647827+0000 front 2024-01-24T14:25:40.647938+0000 (oldest deadline 2024-01-24T14:26:05.347799+0000) 2024-01-24T14:26:10.707 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:10 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:10.370+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:45.348138+0000 front 2024-01-24T14:25:45.348156+0000 (oldest deadline 2024-01-24T14:26:08.848352+0000) 2024-01-24T14:26:10.735 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:26:10 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6[76344]: 2024-01-24T14:26:10.311+0000 7f6f362bd700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.007528+0000 front 2024-01-24T14:25:40.007600+0000 (oldest deadline 2024-01-24T14:26:04.107485+0000) 2024-01-24T14:26:10.736 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:26:10 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6[76344]: 2024-01-24T14:26:10.311+0000 7f6f362bd700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:44.108101+0000 front 2024-01-24T14:25:44.108001+0000 (oldest deadline 2024-01-24T14:26:10.008140+0000) 2024-01-24T14:26:10.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:10 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:10.411+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.092357+0000 front 2024-01-24T14:25:38.092298+0000 (oldest deadline 2024-01-24T14:26:03.992419+0000) 2024-01-24T14:26:10.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:10 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:10.411+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:43.993283+0000 front 2024-01-24T14:25:43.992896+0000 (oldest deadline 2024-01-24T14:26:09.892936+0000) 2024-01-24T14:26:11.094 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jan 24 14:26:10 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-5[59863]: 2024-01-24T14:26:10.866+0000 7f9971fc7700 -1 osd.5 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:36.584653+0000 front 2024-01-24T14:25:36.584717+0000 (oldest deadline 2024-01-24T14:26:02.484505+0000) 2024-01-24T14:26:11.094 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:10 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:10.670+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:11.094 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:10 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:10.670+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:11.415 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@osd.4.service' 2024-01-24T14:26:11.456 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T14:26:11.457 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-01-24T14:26:11.457 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-01-24T14:26:11.457 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@osd.5 2024-01-24T14:26:11.487 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:11 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:11.235+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:39.060282+0000 front 2024-01-24T14:25:39.060365+0000 (oldest deadline 2024-01-24T14:26:02.560180+0000) 2024-01-24T14:26:11.487 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:11 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:11.235+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:47.261420+0000 front 2024-01-24T14:25:47.261437+0000 (oldest deadline 2024-01-24T14:26:10.161656+0000) 2024-01-24T14:26:11.736 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:26:11 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6[76344]: 2024-01-24T14:26:11.269+0000 7f6f362bd700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.007528+0000 front 2024-01-24T14:25:40.007600+0000 (oldest deadline 2024-01-24T14:26:04.107485+0000) 2024-01-24T14:26:11.737 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:26:11 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6[76344]: 2024-01-24T14:26:11.269+0000 7f6f362bd700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:44.108101+0000 front 2024-01-24T14:25:44.108001+0000 (oldest deadline 2024-01-24T14:26:10.008140+0000) 2024-01-24T14:26:11.737 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:11 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:11.395+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.092357+0000 front 2024-01-24T14:25:38.092298+0000 (oldest deadline 2024-01-24T14:26:03.992419+0000) 2024-01-24T14:26:11.737 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:11 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:11.395+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:43.993283+0000 front 2024-01-24T14:25:43.992896+0000 (oldest deadline 2024-01-24T14:26:09.892936+0000) 2024-01-24T14:26:11.799 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:11 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:11.355+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.647827+0000 front 2024-01-24T14:25:40.647938+0000 (oldest deadline 2024-01-24T14:26:05.347799+0000) 2024-01-24T14:26:11.799 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:11 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:11.355+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:45.348138+0000 front 2024-01-24T14:25:45.348156+0000 (oldest deadline 2024-01-24T14:26:08.848352+0000) 2024-01-24T14:26:11.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:11 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:11.691+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:11.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:11 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:11.691+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:11.844 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jan 24 14:26:11 smithi194 systemd[1]: Stopping Ceph osd.5 for 654b0c8e-bac3-11ee-95b1-87774f69a715... 2024-01-24T14:26:11.844 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jan 24 14:26:11 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-5[59863]: 2024-01-24T14:26:11.701+0000 7f99791d8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T14:26:11.844 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jan 24 14:26:11 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-5[59863]: 2024-01-24T14:26:11.701+0000 7f99791d8700 -1 osd.5 82 *** Got signal Terminated *** 2024-01-24T14:26:11.844 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jan 24 14:26:11 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-5[59863]: 2024-01-24T14:26:11.701+0000 7f99791d8700 -1 osd.5 82 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-01-24T14:26:12.191 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jan 24 14:26:11 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-5[59863]: 2024-01-24T14:26:11.915+0000 7f9971fc7700 -1 osd.5 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:36.584653+0000 front 2024-01-24T14:25:36.584717+0000 (oldest deadline 2024-01-24T14:26:02.484505+0000) 2024-01-24T14:26:12.191 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jan 24 14:26:11 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-5[59863]: 2024-01-24T14:26:11.915+0000 7f9971fc7700 -1 osd.5 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.287238+0000 front 2024-01-24T14:25:46.287283+0000 (oldest deadline 2024-01-24T14:26:10.987245+0000) 2024-01-24T14:26:12.594 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:12 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:12.190+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:39.060282+0000 front 2024-01-24T14:25:39.060365+0000 (oldest deadline 2024-01-24T14:26:02.560180+0000) 2024-01-24T14:26:12.594 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:12 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:12.190+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:47.261420+0000 front 2024-01-24T14:25:47.261437+0000 (oldest deadline 2024-01-24T14:26:10.161656+0000) 2024-01-24T14:26:12.736 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:26:12 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6[76344]: 2024-01-24T14:26:12.273+0000 7f6f362bd700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.007528+0000 front 2024-01-24T14:25:40.007600+0000 (oldest deadline 2024-01-24T14:26:04.107485+0000) 2024-01-24T14:26:12.736 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:26:12 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6[76344]: 2024-01-24T14:26:12.273+0000 7f6f362bd700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:44.108101+0000 front 2024-01-24T14:25:44.108001+0000 (oldest deadline 2024-01-24T14:26:10.008140+0000) 2024-01-24T14:26:12.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:12 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:12.374+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.092357+0000 front 2024-01-24T14:25:38.092298+0000 (oldest deadline 2024-01-24T14:26:03.992419+0000) 2024-01-24T14:26:12.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:12 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:12.374+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:43.993283+0000 front 2024-01-24T14:25:43.992896+0000 (oldest deadline 2024-01-24T14:26:09.892936+0000) 2024-01-24T14:26:12.799 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:12 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:12.397+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.647827+0000 front 2024-01-24T14:25:40.647938+0000 (oldest deadline 2024-01-24T14:26:05.347799+0000) 2024-01-24T14:26:12.799 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:12 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:12.397+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:45.348138+0000 front 2024-01-24T14:25:45.348156+0000 (oldest deadline 2024-01-24T14:26:08.848352+0000) 2024-01-24T14:26:12.915 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:12 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:12.659+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:12.915 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:12 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:12.659+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:13.344 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jan 24 14:26:12 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-5[59863]: 2024-01-24T14:26:12.914+0000 7f9971fc7700 -1 osd.5 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:36.584653+0000 front 2024-01-24T14:25:36.584717+0000 (oldest deadline 2024-01-24T14:26:02.484505+0000) 2024-01-24T14:26:13.344 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jan 24 14:26:12 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-5[59863]: 2024-01-24T14:26:12.914+0000 7f9971fc7700 -1 osd.5 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.287238+0000 front 2024-01-24T14:25:46.287283+0000 (oldest deadline 2024-01-24T14:26:10.987245+0000) 2024-01-24T14:26:13.344 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:13 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:13.161+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:39.060282+0000 front 2024-01-24T14:25:39.060365+0000 (oldest deadline 2024-01-24T14:26:02.560180+0000) 2024-01-24T14:26:13.345 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:13 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:13.161+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:47.261420+0000 front 2024-01-24T14:25:47.261437+0000 (oldest deadline 2024-01-24T14:26:10.161656+0000) 2024-01-24T14:26:13.735 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:26:13 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6[76344]: 2024-01-24T14:26:13.286+0000 7f6f362bd700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.007528+0000 front 2024-01-24T14:25:40.007600+0000 (oldest deadline 2024-01-24T14:26:04.107485+0000) 2024-01-24T14:26:13.736 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:26:13 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6[76344]: 2024-01-24T14:26:13.286+0000 7f6f362bd700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:44.108101+0000 front 2024-01-24T14:25:44.108001+0000 (oldest deadline 2024-01-24T14:26:10.008140+0000) 2024-01-24T14:26:13.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:13 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:13.346+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.092357+0000 front 2024-01-24T14:25:38.092298+0000 (oldest deadline 2024-01-24T14:26:03.992419+0000) 2024-01-24T14:26:13.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:13 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:13.346+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:43.993283+0000 front 2024-01-24T14:25:43.992896+0000 (oldest deadline 2024-01-24T14:26:09.892936+0000) 2024-01-24T14:26:13.799 INFO:journalctl@ceph.osd.7.smithi123.stdout:Jan 24 14:26:13 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-7[66344]: 2024-01-24T14:26:13.403+0000 7fd8d13e1700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:47.573271+0000 front 2024-01-24T14:25:47.573290+0000 (oldest deadline 2024-01-24T14:26:12.873464+0000) 2024-01-24T14:26:13.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:13 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:13.402+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.647827+0000 front 2024-01-24T14:25:40.647938+0000 (oldest deadline 2024-01-24T14:26:05.347799+0000) 2024-01-24T14:26:13.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:13 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:13.402+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:45.348138+0000 front 2024-01-24T14:25:45.348156+0000 (oldest deadline 2024-01-24T14:26:08.848352+0000) 2024-01-24T14:26:13.949 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:13 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:13.674+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:13.949 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:13 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:13.674+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:14.344 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jan 24 14:26:13 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-5[59863]: 2024-01-24T14:26:13.948+0000 7f9971fc7700 -1 osd.5 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:36.584653+0000 front 2024-01-24T14:25:36.584717+0000 (oldest deadline 2024-01-24T14:26:02.484505+0000) 2024-01-24T14:26:14.344 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jan 24 14:26:13 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-5[59863]: 2024-01-24T14:26:13.948+0000 7f9971fc7700 -1 osd.5 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.287238+0000 front 2024-01-24T14:25:46.287283+0000 (oldest deadline 2024-01-24T14:26:10.987245+0000) 2024-01-24T14:26:14.344 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:14 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:14.161+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:39.060282+0000 front 2024-01-24T14:25:39.060365+0000 (oldest deadline 2024-01-24T14:26:02.560180+0000) 2024-01-24T14:26:14.344 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:14 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:14.161+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:47.261420+0000 front 2024-01-24T14:25:47.261437+0000 (oldest deadline 2024-01-24T14:26:10.161656+0000) 2024-01-24T14:26:14.736 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:26:14 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6[76344]: 2024-01-24T14:26:14.264+0000 7f6f362bd700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.007528+0000 front 2024-01-24T14:25:40.007600+0000 (oldest deadline 2024-01-24T14:26:04.107485+0000) 2024-01-24T14:26:14.736 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:26:14 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6[76344]: 2024-01-24T14:26:14.264+0000 7f6f362bd700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:44.108101+0000 front 2024-01-24T14:25:44.108001+0000 (oldest deadline 2024-01-24T14:26:10.008140+0000) 2024-01-24T14:26:14.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:14 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:14.299+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.092357+0000 front 2024-01-24T14:25:38.092298+0000 (oldest deadline 2024-01-24T14:26:03.992419+0000) 2024-01-24T14:26:14.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:14 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:14.299+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:43.993283+0000 front 2024-01-24T14:25:43.992896+0000 (oldest deadline 2024-01-24T14:26:09.892936+0000) 2024-01-24T14:26:14.799 INFO:journalctl@ceph.osd.7.smithi123.stdout:Jan 24 14:26:14 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-7[66344]: 2024-01-24T14:26:14.426+0000 7fd8d13e1700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:47.573271+0000 front 2024-01-24T14:25:47.573290+0000 (oldest deadline 2024-01-24T14:26:12.873464+0000) 2024-01-24T14:26:14.799 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:14 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:14.380+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.647827+0000 front 2024-01-24T14:25:40.647938+0000 (oldest deadline 2024-01-24T14:26:05.347799+0000) 2024-01-24T14:26:14.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:14 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:14.380+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:45.348138+0000 front 2024-01-24T14:25:45.348156+0000 (oldest deadline 2024-01-24T14:26:08.848352+0000) 2024-01-24T14:26:14.981 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:14 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:14.680+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:14.981 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:14 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:14.680+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:15.344 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jan 24 14:26:14 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-5[59863]: 2024-01-24T14:26:14.980+0000 7f9971fc7700 -1 osd.5 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:36.584653+0000 front 2024-01-24T14:25:36.584717+0000 (oldest deadline 2024-01-24T14:26:02.484505+0000) 2024-01-24T14:26:15.344 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jan 24 14:26:14 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-5[59863]: 2024-01-24T14:26:14.980+0000 7f9971fc7700 -1 osd.5 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.287238+0000 front 2024-01-24T14:25:46.287283+0000 (oldest deadline 2024-01-24T14:26:10.987245+0000) 2024-01-24T14:26:15.344 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:15 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:15.139+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:39.060282+0000 front 2024-01-24T14:25:39.060365+0000 (oldest deadline 2024-01-24T14:26:02.560180+0000) 2024-01-24T14:26:15.344 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:15 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:15.139+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:47.261420+0000 front 2024-01-24T14:25:47.261437+0000 (oldest deadline 2024-01-24T14:26:10.161656+0000) 2024-01-24T14:26:15.735 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:26:15 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6[76344]: 2024-01-24T14:26:15.257+0000 7f6f362bd700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.007528+0000 front 2024-01-24T14:25:40.007600+0000 (oldest deadline 2024-01-24T14:26:04.107485+0000) 2024-01-24T14:26:15.736 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:26:15 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6[76344]: 2024-01-24T14:26:15.257+0000 7f6f362bd700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:44.108101+0000 front 2024-01-24T14:25:44.108001+0000 (oldest deadline 2024-01-24T14:26:10.008140+0000) 2024-01-24T14:26:15.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:15 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:15.263+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.092357+0000 front 2024-01-24T14:25:38.092298+0000 (oldest deadline 2024-01-24T14:26:03.992419+0000) 2024-01-24T14:26:15.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:15 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:15.263+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:43.993283+0000 front 2024-01-24T14:25:43.992896+0000 (oldest deadline 2024-01-24T14:26:09.892936+0000) 2024-01-24T14:26:15.799 INFO:journalctl@ceph.osd.7.smithi123.stdout:Jan 24 14:26:15 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-7[66344]: 2024-01-24T14:26:15.449+0000 7fd8d13e1700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:47.573271+0000 front 2024-01-24T14:25:47.573290+0000 (oldest deadline 2024-01-24T14:26:12.873464+0000) 2024-01-24T14:26:15.799 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:15 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:15.421+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.647827+0000 front 2024-01-24T14:25:40.647938+0000 (oldest deadline 2024-01-24T14:26:05.347799+0000) 2024-01-24T14:26:15.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:15 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:15.421+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:45.348138+0000 front 2024-01-24T14:25:45.348156+0000 (oldest deadline 2024-01-24T14:26:08.848352+0000) 2024-01-24T14:26:15.964 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:15 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:15.643+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:15.964 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:15 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:15.643+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:16.344 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jan 24 14:26:15 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-5[59863]: 2024-01-24T14:26:15.963+0000 7f9971fc7700 -1 osd.5 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:36.584653+0000 front 2024-01-24T14:25:36.584717+0000 (oldest deadline 2024-01-24T14:26:02.484505+0000) 2024-01-24T14:26:16.344 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jan 24 14:26:15 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-5[59863]: 2024-01-24T14:26:15.963+0000 7f9971fc7700 -1 osd.5 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.287238+0000 front 2024-01-24T14:25:46.287283+0000 (oldest deadline 2024-01-24T14:26:10.987245+0000) 2024-01-24T14:26:16.344 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:16 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:16.155+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:39.060282+0000 front 2024-01-24T14:25:39.060365+0000 (oldest deadline 2024-01-24T14:26:02.560180+0000) 2024-01-24T14:26:16.344 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:16 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:16.156+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:47.261420+0000 front 2024-01-24T14:25:47.261437+0000 (oldest deadline 2024-01-24T14:26:10.161656+0000) 2024-01-24T14:26:16.736 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:26:16 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6[76344]: 2024-01-24T14:26:16.275+0000 7f6f362bd700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.007528+0000 front 2024-01-24T14:25:40.007600+0000 (oldest deadline 2024-01-24T14:26:04.107485+0000) 2024-01-24T14:26:16.736 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:26:16 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6[76344]: 2024-01-24T14:26:16.275+0000 7f6f362bd700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:44.108101+0000 front 2024-01-24T14:25:44.108001+0000 (oldest deadline 2024-01-24T14:26:10.008140+0000) 2024-01-24T14:26:16.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:16 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:16.285+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.092357+0000 front 2024-01-24T14:25:38.092298+0000 (oldest deadline 2024-01-24T14:26:03.992419+0000) 2024-01-24T14:26:16.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:16 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:16.285+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:43.993283+0000 front 2024-01-24T14:25:43.992896+0000 (oldest deadline 2024-01-24T14:26:09.892936+0000) 2024-01-24T14:26:16.799 INFO:journalctl@ceph.osd.7.smithi123.stdout:Jan 24 14:26:16 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-7[66344]: 2024-01-24T14:26:16.439+0000 7fd8d13e1700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:47.573271+0000 front 2024-01-24T14:25:47.573290+0000 (oldest deadline 2024-01-24T14:26:12.873464+0000) 2024-01-24T14:26:16.799 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:16 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:16.439+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.647827+0000 front 2024-01-24T14:25:40.647938+0000 (oldest deadline 2024-01-24T14:26:05.347799+0000) 2024-01-24T14:26:16.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:16 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:16.439+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:45.348138+0000 front 2024-01-24T14:25:45.348156+0000 (oldest deadline 2024-01-24T14:26:08.848352+0000) 2024-01-24T14:26:17.094 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:16 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:16.609+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:17.094 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:16 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:16.609+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:17.594 INFO:journalctl@ceph.osd.5.smithi194.stdout:Jan 24 14:26:17 smithi194 bash[85025]: ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-5 2024-01-24T14:26:17.594 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:17 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:17.191+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:39.060282+0000 front 2024-01-24T14:25:39.060365+0000 (oldest deadline 2024-01-24T14:26:02.560180+0000) 2024-01-24T14:26:17.594 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:17 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:17.191+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:47.261420+0000 front 2024-01-24T14:25:47.261437+0000 (oldest deadline 2024-01-24T14:26:10.161656+0000) 2024-01-24T14:26:17.736 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:26:17 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6[76344]: 2024-01-24T14:26:17.282+0000 7f6f362bd700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.007528+0000 front 2024-01-24T14:25:40.007600+0000 (oldest deadline 2024-01-24T14:26:04.107485+0000) 2024-01-24T14:26:17.737 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:26:17 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6[76344]: 2024-01-24T14:26:17.282+0000 7f6f362bd700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:44.108101+0000 front 2024-01-24T14:25:44.108001+0000 (oldest deadline 2024-01-24T14:26:10.008140+0000) 2024-01-24T14:26:17.737 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:17 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:17.288+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.092357+0000 front 2024-01-24T14:25:38.092298+0000 (oldest deadline 2024-01-24T14:26:03.992419+0000) 2024-01-24T14:26:17.737 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:17 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:17.288+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:43.993283+0000 front 2024-01-24T14:25:43.992896+0000 (oldest deadline 2024-01-24T14:26:09.892936+0000) 2024-01-24T14:26:17.799 INFO:journalctl@ceph.osd.7.smithi123.stdout:Jan 24 14:26:17 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-7[66344]: 2024-01-24T14:26:17.474+0000 7fd8d13e1700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:47.573271+0000 front 2024-01-24T14:25:47.573290+0000 (oldest deadline 2024-01-24T14:26:12.873464+0000) 2024-01-24T14:26:17.799 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:17 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:17.432+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.647827+0000 front 2024-01-24T14:25:40.647938+0000 (oldest deadline 2024-01-24T14:26:05.347799+0000) 2024-01-24T14:26:17.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:17 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:17.432+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:45.348138+0000 front 2024-01-24T14:25:45.348156+0000 (oldest deadline 2024-01-24T14:26:08.848352+0000) 2024-01-24T14:26:18.024 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:17 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:17.656+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:18.024 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:17 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:17.656+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:18.024 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:17 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:17.656+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.424574+0000 front 2024-01-24T14:25:50.424437+0000 (oldest deadline 2024-01-24T14:26:17.425233+0000) 2024-01-24T14:26:18.344 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:18 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:18.173+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:39.060282+0000 front 2024-01-24T14:25:39.060365+0000 (oldest deadline 2024-01-24T14:26:02.560180+0000) 2024-01-24T14:26:18.344 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:18 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:18.173+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:47.261420+0000 front 2024-01-24T14:25:47.261437+0000 (oldest deadline 2024-01-24T14:26:10.161656+0000) 2024-01-24T14:26:18.356 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@osd.5.service' 2024-01-24T14:26:18.439 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T14:26:18.439 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-01-24T14:26:18.440 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-01-24T14:26:18.440 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl stop ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@osd.6 2024-01-24T14:26:18.696 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:18 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:18.285+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.092357+0000 front 2024-01-24T14:25:38.092298+0000 (oldest deadline 2024-01-24T14:26:03.992419+0000) 2024-01-24T14:26:18.696 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:18 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:18.285+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:43.993283+0000 front 2024-01-24T14:25:43.992896+0000 (oldest deadline 2024-01-24T14:26:09.892936+0000) 2024-01-24T14:26:18.696 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:18 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:18.285+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.993828+0000 front 2024-01-24T14:25:50.993806+0000 (oldest deadline 2024-01-24T14:26:17.394356+0000) 2024-01-24T14:26:18.696 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:26:18 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6[76344]: 2024-01-24T14:26:18.290+0000 7f6f362bd700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.007528+0000 front 2024-01-24T14:25:40.007600+0000 (oldest deadline 2024-01-24T14:26:04.107485+0000) 2024-01-24T14:26:18.696 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:26:18 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6[76344]: 2024-01-24T14:26:18.290+0000 7f6f362bd700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:44.108101+0000 front 2024-01-24T14:25:44.108001+0000 (oldest deadline 2024-01-24T14:26:10.008140+0000) 2024-01-24T14:26:18.697 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:26:18 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6[76344]: 2024-01-24T14:26:18.290+0000 7f6f362bd700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:52.309433+0000 front 2024-01-24T14:25:52.309497+0000 (oldest deadline 2024-01-24T14:26:18.209437+0000) 2024-01-24T14:26:18.697 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:26:18 smithi019 systemd[1]: Stopping Ceph osd.6 for 654b0c8e-bac3-11ee-95b1-87774f69a715... 2024-01-24T14:26:18.799 INFO:journalctl@ceph.osd.7.smithi123.stdout:Jan 24 14:26:18 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-7[66344]: 2024-01-24T14:26:18.479+0000 7fd8d13e1700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:47.573271+0000 front 2024-01-24T14:25:47.573290+0000 (oldest deadline 2024-01-24T14:26:12.873464+0000) 2024-01-24T14:26:18.799 INFO:journalctl@ceph.osd.7.smithi123.stdout:Jan 24 14:26:18 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-7[66344]: 2024-01-24T14:26:18.479+0000 7fd8d13e1700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:55.174641+0000 front 2024-01-24T14:25:55.174696+0000 (oldest deadline 2024-01-24T14:26:18.074488+0000) 2024-01-24T14:26:18.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:18 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:18.413+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.647827+0000 front 2024-01-24T14:25:40.647938+0000 (oldest deadline 2024-01-24T14:26:05.347799+0000) 2024-01-24T14:26:18.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:18 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:18.413+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:45.348138+0000 front 2024-01-24T14:25:45.348156+0000 (oldest deadline 2024-01-24T14:26:08.848352+0000) 2024-01-24T14:26:18.985 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:26:18 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6[76344]: 2024-01-24T14:26:18.694+0000 7f6f3dccf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T14:26:18.985 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:26:18 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6[76344]: 2024-01-24T14:26:18.694+0000 7f6f3dccf700 -1 osd.6 82 *** Got signal Terminated *** 2024-01-24T14:26:18.986 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:26:18 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6[76344]: 2024-01-24T14:26:18.694+0000 7f6f3dccf700 -1 osd.6 82 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-01-24T14:26:19.094 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:18 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:18.689+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:19.094 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:18 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:18.689+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:19.094 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:18 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:18.689+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.424574+0000 front 2024-01-24T14:25:50.424437+0000 (oldest deadline 2024-01-24T14:26:17.425233+0000) 2024-01-24T14:26:19.594 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:19 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:19.186+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:39.060282+0000 front 2024-01-24T14:25:39.060365+0000 (oldest deadline 2024-01-24T14:26:02.560180+0000) 2024-01-24T14:26:19.594 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:19 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:19.186+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:47.261420+0000 front 2024-01-24T14:25:47.261437+0000 (oldest deadline 2024-01-24T14:26:10.161656+0000) 2024-01-24T14:26:19.737 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:26:19 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6[76344]: 2024-01-24T14:26:19.308+0000 7f6f362bd700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.007528+0000 front 2024-01-24T14:25:40.007600+0000 (oldest deadline 2024-01-24T14:26:04.107485+0000) 2024-01-24T14:26:19.738 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:26:19 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6[76344]: 2024-01-24T14:26:19.308+0000 7f6f362bd700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:44.108101+0000 front 2024-01-24T14:25:44.108001+0000 (oldest deadline 2024-01-24T14:26:10.008140+0000) 2024-01-24T14:26:19.738 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:26:19 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6[76344]: 2024-01-24T14:26:19.308+0000 7f6f362bd700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:52.309433+0000 front 2024-01-24T14:25:52.309497+0000 (oldest deadline 2024-01-24T14:26:18.209437+0000) 2024-01-24T14:26:19.738 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:19 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:19.270+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.092357+0000 front 2024-01-24T14:25:38.092298+0000 (oldest deadline 2024-01-24T14:26:03.992419+0000) 2024-01-24T14:26:19.738 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:19 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:19.270+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:43.993283+0000 front 2024-01-24T14:25:43.992896+0000 (oldest deadline 2024-01-24T14:26:09.892936+0000) 2024-01-24T14:26:19.738 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:19 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:19.270+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.993828+0000 front 2024-01-24T14:25:50.993806+0000 (oldest deadline 2024-01-24T14:26:17.394356+0000) 2024-01-24T14:26:19.802 INFO:journalctl@ceph.osd.7.smithi123.stdout:Jan 24 14:26:19 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-7[66344]: 2024-01-24T14:26:19.514+0000 7fd8d13e1700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:47.573271+0000 front 2024-01-24T14:25:47.573290+0000 (oldest deadline 2024-01-24T14:26:12.873464+0000) 2024-01-24T14:26:19.802 INFO:journalctl@ceph.osd.7.smithi123.stdout:Jan 24 14:26:19 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-7[66344]: 2024-01-24T14:26:19.514+0000 7fd8d13e1700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:55.174641+0000 front 2024-01-24T14:25:55.174696+0000 (oldest deadline 2024-01-24T14:26:18.074488+0000) 2024-01-24T14:26:19.803 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:19 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:19.428+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.647827+0000 front 2024-01-24T14:25:40.647938+0000 (oldest deadline 2024-01-24T14:26:05.347799+0000) 2024-01-24T14:26:19.803 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:19 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:19.428+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:45.348138+0000 front 2024-01-24T14:25:45.348156+0000 (oldest deadline 2024-01-24T14:26:08.848352+0000) 2024-01-24T14:26:19.803 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:19 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:19.428+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.749398+0000 front 2024-01-24T14:25:54.749444+0000 (oldest deadline 2024-01-24T14:26:18.849397+0000) 2024-01-24T14:26:20.094 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:19 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:19.664+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:20.094 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:19 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:19.664+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:20.094 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:19 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:19.664+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.424574+0000 front 2024-01-24T14:25:50.424437+0000 (oldest deadline 2024-01-24T14:26:17.425233+0000) 2024-01-24T14:26:20.594 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:20 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:20.151+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:39.060282+0000 front 2024-01-24T14:25:39.060365+0000 (oldest deadline 2024-01-24T14:26:02.560180+0000) 2024-01-24T14:26:20.594 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:20 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:20.151+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:47.261420+0000 front 2024-01-24T14:25:47.261437+0000 (oldest deadline 2024-01-24T14:26:10.161656+0000) 2024-01-24T14:26:20.594 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:20 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:20.152+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.763402+0000 front 2024-01-24T14:25:54.763474+0000 (oldest deadline 2024-01-24T14:26:19.463597+0000) 2024-01-24T14:26:20.735 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:26:20 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6[76344]: 2024-01-24T14:26:20.320+0000 7f6f362bd700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.007528+0000 front 2024-01-24T14:25:40.007600+0000 (oldest deadline 2024-01-24T14:26:04.107485+0000) 2024-01-24T14:26:20.736 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:26:20 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6[76344]: 2024-01-24T14:26:20.320+0000 7f6f362bd700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:44.108101+0000 front 2024-01-24T14:25:44.108001+0000 (oldest deadline 2024-01-24T14:26:10.008140+0000) 2024-01-24T14:26:20.736 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:26:20 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6[76344]: 2024-01-24T14:26:20.320+0000 7f6f362bd700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:52.309433+0000 front 2024-01-24T14:25:52.309497+0000 (oldest deadline 2024-01-24T14:26:18.209437+0000) 2024-01-24T14:26:20.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:20 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:20.280+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.092357+0000 front 2024-01-24T14:25:38.092298+0000 (oldest deadline 2024-01-24T14:26:03.992419+0000) 2024-01-24T14:26:20.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:20 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:20.280+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:43.993283+0000 front 2024-01-24T14:25:43.992896+0000 (oldest deadline 2024-01-24T14:26:09.892936+0000) 2024-01-24T14:26:20.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:20 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:20.280+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.993828+0000 front 2024-01-24T14:25:50.993806+0000 (oldest deadline 2024-01-24T14:26:17.394356+0000) 2024-01-24T14:26:20.799 INFO:journalctl@ceph.osd.7.smithi123.stdout:Jan 24 14:26:20 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-7[66344]: 2024-01-24T14:26:20.502+0000 7fd8d13e1700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:47.573271+0000 front 2024-01-24T14:25:47.573290+0000 (oldest deadline 2024-01-24T14:26:12.873464+0000) 2024-01-24T14:26:20.799 INFO:journalctl@ceph.osd.7.smithi123.stdout:Jan 24 14:26:20 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-7[66344]: 2024-01-24T14:26:20.502+0000 7fd8d13e1700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:55.174641+0000 front 2024-01-24T14:25:55.174696+0000 (oldest deadline 2024-01-24T14:26:18.074488+0000) 2024-01-24T14:26:20.799 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:20 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:20.388+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.647827+0000 front 2024-01-24T14:25:40.647938+0000 (oldest deadline 2024-01-24T14:26:05.347799+0000) 2024-01-24T14:26:20.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:20 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:20.388+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:45.348138+0000 front 2024-01-24T14:25:45.348156+0000 (oldest deadline 2024-01-24T14:26:08.848352+0000) 2024-01-24T14:26:20.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:20 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:20.388+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.749398+0000 front 2024-01-24T14:25:54.749444+0000 (oldest deadline 2024-01-24T14:26:18.849397+0000) 2024-01-24T14:26:21.096 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:20 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:20.626+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:21.096 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:20 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:20.626+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:21.096 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:20 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:20.626+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.424574+0000 front 2024-01-24T14:25:50.424437+0000 (oldest deadline 2024-01-24T14:26:17.425233+0000) 2024-01-24T14:26:21.594 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:21 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:21.115+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:39.060282+0000 front 2024-01-24T14:25:39.060365+0000 (oldest deadline 2024-01-24T14:26:02.560180+0000) 2024-01-24T14:26:21.594 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:21 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:21.115+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:47.261420+0000 front 2024-01-24T14:25:47.261437+0000 (oldest deadline 2024-01-24T14:26:10.161656+0000) 2024-01-24T14:26:21.594 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:21 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:21.115+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.763402+0000 front 2024-01-24T14:25:54.763474+0000 (oldest deadline 2024-01-24T14:26:19.463597+0000) 2024-01-24T14:26:21.735 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:21 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:21.262+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.092357+0000 front 2024-01-24T14:25:38.092298+0000 (oldest deadline 2024-01-24T14:26:03.992419+0000) 2024-01-24T14:26:21.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:21 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:21.262+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:43.993283+0000 front 2024-01-24T14:25:43.992896+0000 (oldest deadline 2024-01-24T14:26:09.892936+0000) 2024-01-24T14:26:21.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:21 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:21.262+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.993828+0000 front 2024-01-24T14:25:50.993806+0000 (oldest deadline 2024-01-24T14:26:17.394356+0000) 2024-01-24T14:26:21.736 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:26:21 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6[76344]: 2024-01-24T14:26:21.287+0000 7f6f362bd700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.007528+0000 front 2024-01-24T14:25:40.007600+0000 (oldest deadline 2024-01-24T14:26:04.107485+0000) 2024-01-24T14:26:21.736 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:26:21 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6[76344]: 2024-01-24T14:26:21.287+0000 7f6f362bd700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:44.108101+0000 front 2024-01-24T14:25:44.108001+0000 (oldest deadline 2024-01-24T14:26:10.008140+0000) 2024-01-24T14:26:21.736 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:26:21 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6[76344]: 2024-01-24T14:26:21.287+0000 7f6f362bd700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:52.309433+0000 front 2024-01-24T14:25:52.309497+0000 (oldest deadline 2024-01-24T14:26:18.209437+0000) 2024-01-24T14:26:21.799 INFO:journalctl@ceph.osd.7.smithi123.stdout:Jan 24 14:26:21 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-7[66344]: 2024-01-24T14:26:21.457+0000 7fd8d13e1700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:47.573271+0000 front 2024-01-24T14:25:47.573290+0000 (oldest deadline 2024-01-24T14:26:12.873464+0000) 2024-01-24T14:26:21.799 INFO:journalctl@ceph.osd.7.smithi123.stdout:Jan 24 14:26:21 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-7[66344]: 2024-01-24T14:26:21.457+0000 7fd8d13e1700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:55.174641+0000 front 2024-01-24T14:25:55.174696+0000 (oldest deadline 2024-01-24T14:26:18.074488+0000) 2024-01-24T14:26:21.799 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:21 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:21.384+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.647827+0000 front 2024-01-24T14:25:40.647938+0000 (oldest deadline 2024-01-24T14:26:05.347799+0000) 2024-01-24T14:26:21.799 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:21 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:21.384+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:45.348138+0000 front 2024-01-24T14:25:45.348156+0000 (oldest deadline 2024-01-24T14:26:08.848352+0000) 2024-01-24T14:26:21.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:21 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:21.384+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.749398+0000 front 2024-01-24T14:25:54.749444+0000 (oldest deadline 2024-01-24T14:26:18.849397+0000) 2024-01-24T14:26:22.094 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:21 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:21.662+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:22.094 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:21 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:21.662+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:22.094 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:21 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:21.662+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.424574+0000 front 2024-01-24T14:25:50.424437+0000 (oldest deadline 2024-01-24T14:26:17.425233+0000) 2024-01-24T14:26:22.594 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:22 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:22.114+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:39.060282+0000 front 2024-01-24T14:25:39.060365+0000 (oldest deadline 2024-01-24T14:26:02.560180+0000) 2024-01-24T14:26:22.594 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:22 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:22.114+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:47.261420+0000 front 2024-01-24T14:25:47.261437+0000 (oldest deadline 2024-01-24T14:26:10.161656+0000) 2024-01-24T14:26:22.594 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:22 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:22.114+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.763402+0000 front 2024-01-24T14:25:54.763474+0000 (oldest deadline 2024-01-24T14:26:19.463597+0000) 2024-01-24T14:26:22.735 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:26:22 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6[76344]: 2024-01-24T14:26:22.331+0000 7f6f362bd700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.007528+0000 front 2024-01-24T14:25:40.007600+0000 (oldest deadline 2024-01-24T14:26:04.107485+0000) 2024-01-24T14:26:22.736 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:26:22 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6[76344]: 2024-01-24T14:26:22.331+0000 7f6f362bd700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:44.108101+0000 front 2024-01-24T14:25:44.108001+0000 (oldest deadline 2024-01-24T14:26:10.008140+0000) 2024-01-24T14:26:22.736 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:26:22 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6[76344]: 2024-01-24T14:26:22.331+0000 7f6f362bd700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:52.309433+0000 front 2024-01-24T14:25:52.309497+0000 (oldest deadline 2024-01-24T14:26:18.209437+0000) 2024-01-24T14:26:22.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:22 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:22.292+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.092357+0000 front 2024-01-24T14:25:38.092298+0000 (oldest deadline 2024-01-24T14:26:03.992419+0000) 2024-01-24T14:26:22.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:22 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:22.292+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:43.993283+0000 front 2024-01-24T14:25:43.992896+0000 (oldest deadline 2024-01-24T14:26:09.892936+0000) 2024-01-24T14:26:22.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:22 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:22.292+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.993828+0000 front 2024-01-24T14:25:50.993806+0000 (oldest deadline 2024-01-24T14:26:17.394356+0000) 2024-01-24T14:26:22.799 INFO:journalctl@ceph.osd.7.smithi123.stdout:Jan 24 14:26:22 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-7[66344]: 2024-01-24T14:26:22.488+0000 7fd8d13e1700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:47.573271+0000 front 2024-01-24T14:25:47.573290+0000 (oldest deadline 2024-01-24T14:26:12.873464+0000) 2024-01-24T14:26:22.799 INFO:journalctl@ceph.osd.7.smithi123.stdout:Jan 24 14:26:22 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-7[66344]: 2024-01-24T14:26:22.488+0000 7fd8d13e1700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:55.174641+0000 front 2024-01-24T14:25:55.174696+0000 (oldest deadline 2024-01-24T14:26:18.074488+0000) 2024-01-24T14:26:22.799 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:22 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:22.358+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.647827+0000 front 2024-01-24T14:25:40.647938+0000 (oldest deadline 2024-01-24T14:26:05.347799+0000) 2024-01-24T14:26:22.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:22 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:22.358+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:45.348138+0000 front 2024-01-24T14:25:45.348156+0000 (oldest deadline 2024-01-24T14:26:08.848352+0000) 2024-01-24T14:26:22.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:22 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:22.358+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.749398+0000 front 2024-01-24T14:25:54.749444+0000 (oldest deadline 2024-01-24T14:26:18.849397+0000) 2024-01-24T14:26:23.071 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:22 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:22.621+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:23.071 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:22 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:22.621+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:23.071 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:22 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:22.621+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.424574+0000 front 2024-01-24T14:25:50.424437+0000 (oldest deadline 2024-01-24T14:26:17.425233+0000) 2024-01-24T14:26:23.344 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:23 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:23.070+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:39.060282+0000 front 2024-01-24T14:25:39.060365+0000 (oldest deadline 2024-01-24T14:26:02.560180+0000) 2024-01-24T14:26:23.344 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:23 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:23.070+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:47.261420+0000 front 2024-01-24T14:25:47.261437+0000 (oldest deadline 2024-01-24T14:26:10.161656+0000) 2024-01-24T14:26:23.344 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:23 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:23.070+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.763402+0000 front 2024-01-24T14:25:54.763474+0000 (oldest deadline 2024-01-24T14:26:19.463597+0000) 2024-01-24T14:26:23.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:23 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:23.281+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.092357+0000 front 2024-01-24T14:25:38.092298+0000 (oldest deadline 2024-01-24T14:26:03.992419+0000) 2024-01-24T14:26:23.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:23 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:23.281+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:43.993283+0000 front 2024-01-24T14:25:43.992896+0000 (oldest deadline 2024-01-24T14:26:09.892936+0000) 2024-01-24T14:26:23.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:23 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:23.281+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.993828+0000 front 2024-01-24T14:25:50.993806+0000 (oldest deadline 2024-01-24T14:26:17.394356+0000) 2024-01-24T14:26:23.736 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:26:23 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6[76344]: 2024-01-24T14:26:23.358+0000 7f6f362bd700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.007528+0000 front 2024-01-24T14:25:40.007600+0000 (oldest deadline 2024-01-24T14:26:04.107485+0000) 2024-01-24T14:26:23.736 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:26:23 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6[76344]: 2024-01-24T14:26:23.358+0000 7f6f362bd700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:44.108101+0000 front 2024-01-24T14:25:44.108001+0000 (oldest deadline 2024-01-24T14:26:10.008140+0000) 2024-01-24T14:26:23.737 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:26:23 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6[76344]: 2024-01-24T14:26:23.358+0000 7f6f362bd700 -1 osd.6 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:52.309433+0000 front 2024-01-24T14:25:52.309497+0000 (oldest deadline 2024-01-24T14:26:18.209437+0000) 2024-01-24T14:26:23.800 INFO:journalctl@ceph.osd.7.smithi123.stdout:Jan 24 14:26:23 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-7[66344]: 2024-01-24T14:26:23.474+0000 7fd8d13e1700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:47.573271+0000 front 2024-01-24T14:25:47.573290+0000 (oldest deadline 2024-01-24T14:26:12.873464+0000) 2024-01-24T14:26:23.800 INFO:journalctl@ceph.osd.7.smithi123.stdout:Jan 24 14:26:23 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-7[66344]: 2024-01-24T14:26:23.474+0000 7fd8d13e1700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:55.174641+0000 front 2024-01-24T14:25:55.174696+0000 (oldest deadline 2024-01-24T14:26:18.074488+0000) 2024-01-24T14:26:23.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:23 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:23.330+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.647827+0000 front 2024-01-24T14:25:40.647938+0000 (oldest deadline 2024-01-24T14:26:05.347799+0000) 2024-01-24T14:26:23.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:23 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:23.330+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:45.348138+0000 front 2024-01-24T14:25:45.348156+0000 (oldest deadline 2024-01-24T14:26:08.848352+0000) 2024-01-24T14:26:23.801 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:23 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:23.330+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.749398+0000 front 2024-01-24T14:25:54.749444+0000 (oldest deadline 2024-01-24T14:26:18.849397+0000) 2024-01-24T14:26:24.079 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:23 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:23.649+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:24.079 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:23 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:23.649+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:24.079 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:23 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:23.649+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.424574+0000 front 2024-01-24T14:25:50.424437+0000 (oldest deadline 2024-01-24T14:26:17.425233+0000) 2024-01-24T14:26:24.344 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:24 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:24.079+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:39.060282+0000 front 2024-01-24T14:25:39.060365+0000 (oldest deadline 2024-01-24T14:26:02.560180+0000) 2024-01-24T14:26:24.344 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:24 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:24.079+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:47.261420+0000 front 2024-01-24T14:25:47.261437+0000 (oldest deadline 2024-01-24T14:26:10.161656+0000) 2024-01-24T14:26:24.344 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:24 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:24.079+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.763402+0000 front 2024-01-24T14:25:54.763474+0000 (oldest deadline 2024-01-24T14:26:19.463597+0000) 2024-01-24T14:26:24.485 INFO:journalctl@ceph.osd.6.smithi019.stdout:Jan 24 14:26:24 smithi019 bash[98236]: ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-6 2024-01-24T14:26:24.486 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:24 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:24.252+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.092357+0000 front 2024-01-24T14:25:38.092298+0000 (oldest deadline 2024-01-24T14:26:03.992419+0000) 2024-01-24T14:26:24.486 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:24 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:24.252+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:43.993283+0000 front 2024-01-24T14:25:43.992896+0000 (oldest deadline 2024-01-24T14:26:09.892936+0000) 2024-01-24T14:26:24.486 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:24 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:24.252+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.993828+0000 front 2024-01-24T14:25:50.993806+0000 (oldest deadline 2024-01-24T14:26:17.394356+0000) 2024-01-24T14:26:24.799 INFO:journalctl@ceph.osd.7.smithi123.stdout:Jan 24 14:26:24 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-7[66344]: 2024-01-24T14:26:24.467+0000 7fd8d13e1700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:47.573271+0000 front 2024-01-24T14:25:47.573290+0000 (oldest deadline 2024-01-24T14:26:12.873464+0000) 2024-01-24T14:26:24.800 INFO:journalctl@ceph.osd.7.smithi123.stdout:Jan 24 14:26:24 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-7[66344]: 2024-01-24T14:26:24.467+0000 7fd8d13e1700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:55.174641+0000 front 2024-01-24T14:25:55.174696+0000 (oldest deadline 2024-01-24T14:26:18.074488+0000) 2024-01-24T14:26:24.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:24 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:24.368+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.647827+0000 front 2024-01-24T14:25:40.647938+0000 (oldest deadline 2024-01-24T14:26:05.347799+0000) 2024-01-24T14:26:24.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:24 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:24.368+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:45.348138+0000 front 2024-01-24T14:25:45.348156+0000 (oldest deadline 2024-01-24T14:26:08.848352+0000) 2024-01-24T14:26:24.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:24 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:24.368+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.749398+0000 front 2024-01-24T14:25:54.749444+0000 (oldest deadline 2024-01-24T14:26:18.849397+0000) 2024-01-24T14:26:25.094 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:24 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:24.599+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:25.094 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:24 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:24.599+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:25.094 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:24 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:24.599+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.424574+0000 front 2024-01-24T14:25:50.424437+0000 (oldest deadline 2024-01-24T14:26:17.425233+0000) 2024-01-24T14:26:25.392 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:25 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:25.233+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.092357+0000 front 2024-01-24T14:25:38.092298+0000 (oldest deadline 2024-01-24T14:26:03.992419+0000) 2024-01-24T14:26:25.392 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:25 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:25.233+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:43.993283+0000 front 2024-01-24T14:25:43.992896+0000 (oldest deadline 2024-01-24T14:26:09.892936+0000) 2024-01-24T14:26:25.392 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:25 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:25.233+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.993828+0000 front 2024-01-24T14:25:50.993806+0000 (oldest deadline 2024-01-24T14:26:17.394356+0000) 2024-01-24T14:26:25.402 DEBUG:teuthology.orchestra.run.smithi019:> sudo pkill -f 'journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@osd.6.service' 2024-01-24T14:26:25.456 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T14:26:25.456 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-01-24T14:26:25.456 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-01-24T14:26:25.457 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@osd.7 2024-01-24T14:26:25.594 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:25 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:25.115+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:39.060282+0000 front 2024-01-24T14:25:39.060365+0000 (oldest deadline 2024-01-24T14:26:02.560180+0000) 2024-01-24T14:26:25.594 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:25 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:25.115+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:47.261420+0000 front 2024-01-24T14:25:47.261437+0000 (oldest deadline 2024-01-24T14:26:10.161656+0000) 2024-01-24T14:26:25.594 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:25 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:25.115+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.763402+0000 front 2024-01-24T14:25:54.763474+0000 (oldest deadline 2024-01-24T14:26:19.463597+0000) 2024-01-24T14:26:25.699 INFO:journalctl@ceph.osd.7.smithi123.stdout:Jan 24 14:26:25 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-7[66344]: 2024-01-24T14:26:25.428+0000 7fd8d13e1700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:47.573271+0000 front 2024-01-24T14:25:47.573290+0000 (oldest deadline 2024-01-24T14:26:12.873464+0000) 2024-01-24T14:26:25.700 INFO:journalctl@ceph.osd.7.smithi123.stdout:Jan 24 14:26:25 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-7[66344]: 2024-01-24T14:26:25.428+0000 7fd8d13e1700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:55.174641+0000 front 2024-01-24T14:25:55.174696+0000 (oldest deadline 2024-01-24T14:26:18.074488+0000) 2024-01-24T14:26:25.700 INFO:journalctl@ceph.osd.7.smithi123.stdout:Jan 24 14:26:25 smithi123 systemd[1]: Stopping Ceph osd.7 for 654b0c8e-bac3-11ee-95b1-87774f69a715... 2024-01-24T14:26:25.700 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:25 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:25.362+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.647827+0000 front 2024-01-24T14:25:40.647938+0000 (oldest deadline 2024-01-24T14:26:05.347799+0000) 2024-01-24T14:26:25.700 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:25 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:25.362+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:45.348138+0000 front 2024-01-24T14:25:45.348156+0000 (oldest deadline 2024-01-24T14:26:08.848352+0000) 2024-01-24T14:26:25.700 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:25 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:25.362+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.749398+0000 front 2024-01-24T14:25:54.749444+0000 (oldest deadline 2024-01-24T14:26:18.849397+0000) 2024-01-24T14:26:25.700 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:25 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:25.362+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:25:58.849900+0000 front 2024-01-24T14:25:58.850006+0000 (oldest deadline 2024-01-24T14:26:24.749975+0000) 2024-01-24T14:26:26.049 INFO:journalctl@ceph.osd.7.smithi123.stdout:Jan 24 14:26:25 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-7[66344]: 2024-01-24T14:26:25.698+0000 7fd8d5ded700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T14:26:26.049 INFO:journalctl@ceph.osd.7.smithi123.stdout:Jan 24 14:26:25 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-7[66344]: 2024-01-24T14:26:25.698+0000 7fd8d5ded700 -1 osd.7 82 *** Got signal Terminated *** 2024-01-24T14:26:26.049 INFO:journalctl@ceph.osd.7.smithi123.stdout:Jan 24 14:26:25 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-7[66344]: 2024-01-24T14:26:25.698+0000 7fd8d5ded700 -1 osd.7 82 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-01-24T14:26:26.093 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:25 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:25.612+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:26.093 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:25 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:25.612+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:26.094 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:25 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:25.612+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.424574+0000 front 2024-01-24T14:25:50.424437+0000 (oldest deadline 2024-01-24T14:26:17.425233+0000) 2024-01-24T14:26:26.344 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:26 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:26.093+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:39.060282+0000 front 2024-01-24T14:25:39.060365+0000 (oldest deadline 2024-01-24T14:26:02.560180+0000) 2024-01-24T14:26:26.344 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:26 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:26.093+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:47.261420+0000 front 2024-01-24T14:25:47.261437+0000 (oldest deadline 2024-01-24T14:26:10.161656+0000) 2024-01-24T14:26:26.344 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:26 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:26.093+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.763402+0000 front 2024-01-24T14:25:54.763474+0000 (oldest deadline 2024-01-24T14:26:19.463597+0000) 2024-01-24T14:26:26.344 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:26 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:26.093+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:26:01.164718+0000 front 2024-01-24T14:26:01.164658+0000 (oldest deadline 2024-01-24T14:26:25.264501+0000) 2024-01-24T14:26:26.735 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:26 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:26.258+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.092357+0000 front 2024-01-24T14:25:38.092298+0000 (oldest deadline 2024-01-24T14:26:03.992419+0000) 2024-01-24T14:26:26.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:26 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:26.258+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:43.993283+0000 front 2024-01-24T14:25:43.992896+0000 (oldest deadline 2024-01-24T14:26:09.892936+0000) 2024-01-24T14:26:26.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:26 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:26.258+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.993828+0000 front 2024-01-24T14:25:50.993806+0000 (oldest deadline 2024-01-24T14:26:17.394356+0000) 2024-01-24T14:26:26.799 INFO:journalctl@ceph.osd.7.smithi123.stdout:Jan 24 14:26:26 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-7[66344]: 2024-01-24T14:26:26.390+0000 7fd8d13e1700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:47.573271+0000 front 2024-01-24T14:25:47.573290+0000 (oldest deadline 2024-01-24T14:26:12.873464+0000) 2024-01-24T14:26:26.799 INFO:journalctl@ceph.osd.7.smithi123.stdout:Jan 24 14:26:26 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-7[66344]: 2024-01-24T14:26:26.390+0000 7fd8d13e1700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:55.174641+0000 front 2024-01-24T14:25:55.174696+0000 (oldest deadline 2024-01-24T14:26:18.074488+0000) 2024-01-24T14:26:26.799 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:26 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:26.371+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.647827+0000 front 2024-01-24T14:25:40.647938+0000 (oldest deadline 2024-01-24T14:26:05.347799+0000) 2024-01-24T14:26:26.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:26 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:26.371+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:45.348138+0000 front 2024-01-24T14:25:45.348156+0000 (oldest deadline 2024-01-24T14:26:08.848352+0000) 2024-01-24T14:26:26.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:26 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:26.371+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.749398+0000 front 2024-01-24T14:25:54.749444+0000 (oldest deadline 2024-01-24T14:26:18.849397+0000) 2024-01-24T14:26:26.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:26 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:26.371+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:25:58.849900+0000 front 2024-01-24T14:25:58.850006+0000 (oldest deadline 2024-01-24T14:26:24.749975+0000) 2024-01-24T14:26:26.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:26 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:26.564+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:26.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:26 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:26.564+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:26.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:26 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:26.564+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.424574+0000 front 2024-01-24T14:25:50.424437+0000 (oldest deadline 2024-01-24T14:26:17.425233+0000) 2024-01-24T14:26:27.344 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:27 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:27.091+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:39.060282+0000 front 2024-01-24T14:25:39.060365+0000 (oldest deadline 2024-01-24T14:26:02.560180+0000) 2024-01-24T14:26:27.344 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:27 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:27.091+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:47.261420+0000 front 2024-01-24T14:25:47.261437+0000 (oldest deadline 2024-01-24T14:26:10.161656+0000) 2024-01-24T14:26:27.344 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:27 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:27.091+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.763402+0000 front 2024-01-24T14:25:54.763474+0000 (oldest deadline 2024-01-24T14:26:19.463597+0000) 2024-01-24T14:26:27.344 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:27 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:27.091+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:26:01.164718+0000 front 2024-01-24T14:26:01.164658+0000 (oldest deadline 2024-01-24T14:26:25.264501+0000) 2024-01-24T14:26:27.735 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:27 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:27.245+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.092357+0000 front 2024-01-24T14:25:38.092298+0000 (oldest deadline 2024-01-24T14:26:03.992419+0000) 2024-01-24T14:26:27.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:27 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:27.245+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:43.993283+0000 front 2024-01-24T14:25:43.992896+0000 (oldest deadline 2024-01-24T14:26:09.892936+0000) 2024-01-24T14:26:27.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:27 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:27.245+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.993828+0000 front 2024-01-24T14:25:50.993806+0000 (oldest deadline 2024-01-24T14:26:17.394356+0000) 2024-01-24T14:26:27.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:27 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:27.245+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:26:02.496310+0000 front 2024-01-24T14:26:02.496373+0000 (oldest deadline 2024-01-24T14:26:27.096920+0000) 2024-01-24T14:26:27.799 INFO:journalctl@ceph.osd.7.smithi123.stdout:Jan 24 14:26:27 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-7[66344]: 2024-01-24T14:26:27.375+0000 7fd8d13e1700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:47.573271+0000 front 2024-01-24T14:25:47.573290+0000 (oldest deadline 2024-01-24T14:26:12.873464+0000) 2024-01-24T14:26:27.799 INFO:journalctl@ceph.osd.7.smithi123.stdout:Jan 24 14:26:27 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-7[66344]: 2024-01-24T14:26:27.375+0000 7fd8d13e1700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:55.174641+0000 front 2024-01-24T14:25:55.174696+0000 (oldest deadline 2024-01-24T14:26:18.074488+0000) 2024-01-24T14:26:27.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:27 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:27.387+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.647827+0000 front 2024-01-24T14:25:40.647938+0000 (oldest deadline 2024-01-24T14:26:05.347799+0000) 2024-01-24T14:26:27.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:27 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:27.387+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:45.348138+0000 front 2024-01-24T14:25:45.348156+0000 (oldest deadline 2024-01-24T14:26:08.848352+0000) 2024-01-24T14:26:27.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:27 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:27.387+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.749398+0000 front 2024-01-24T14:25:54.749444+0000 (oldest deadline 2024-01-24T14:26:18.849397+0000) 2024-01-24T14:26:27.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:27 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:27.387+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:25:58.849900+0000 front 2024-01-24T14:25:58.850006+0000 (oldest deadline 2024-01-24T14:26:24.749975+0000) 2024-01-24T14:26:27.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:27 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:27.530+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:27.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:27 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:27.530+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:27.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:27 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:27.530+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.424574+0000 front 2024-01-24T14:25:50.424437+0000 (oldest deadline 2024-01-24T14:26:17.425233+0000) 2024-01-24T14:26:28.482 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:28 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:28.120+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:39.060282+0000 front 2024-01-24T14:25:39.060365+0000 (oldest deadline 2024-01-24T14:26:02.560180+0000) 2024-01-24T14:26:28.482 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:28 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:28.120+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:47.261420+0000 front 2024-01-24T14:25:47.261437+0000 (oldest deadline 2024-01-24T14:26:10.161656+0000) 2024-01-24T14:26:28.482 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:28 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:28.120+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.763402+0000 front 2024-01-24T14:25:54.763474+0000 (oldest deadline 2024-01-24T14:26:19.463597+0000) 2024-01-24T14:26:28.482 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:28 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:28.120+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:26:01.164718+0000 front 2024-01-24T14:26:01.164658+0000 (oldest deadline 2024-01-24T14:26:25.264501+0000) 2024-01-24T14:26:28.735 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:28 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:28.271+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.092357+0000 front 2024-01-24T14:25:38.092298+0000 (oldest deadline 2024-01-24T14:26:03.992419+0000) 2024-01-24T14:26:28.735 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:28 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:28.271+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:43.993283+0000 front 2024-01-24T14:25:43.992896+0000 (oldest deadline 2024-01-24T14:26:09.892936+0000) 2024-01-24T14:26:28.735 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:28 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:28.271+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.993828+0000 front 2024-01-24T14:25:50.993806+0000 (oldest deadline 2024-01-24T14:26:17.394356+0000) 2024-01-24T14:26:28.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:28 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:28.271+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:26:02.496310+0000 front 2024-01-24T14:26:02.496373+0000 (oldest deadline 2024-01-24T14:26:27.096920+0000) 2024-01-24T14:26:28.799 INFO:journalctl@ceph.osd.7.smithi123.stdout:Jan 24 14:26:28 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-7[66344]: 2024-01-24T14:26:28.412+0000 7fd8d13e1700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:47.573271+0000 front 2024-01-24T14:25:47.573290+0000 (oldest deadline 2024-01-24T14:26:12.873464+0000) 2024-01-24T14:26:28.799 INFO:journalctl@ceph.osd.7.smithi123.stdout:Jan 24 14:26:28 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-7[66344]: 2024-01-24T14:26:28.412+0000 7fd8d13e1700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:55.174641+0000 front 2024-01-24T14:25:55.174696+0000 (oldest deadline 2024-01-24T14:26:18.074488+0000) 2024-01-24T14:26:28.799 INFO:journalctl@ceph.osd.7.smithi123.stdout:Jan 24 14:26:28 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-7[66344]: 2024-01-24T14:26:28.412+0000 7fd8d13e1700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:25:58.075039+0000 front 2024-01-24T14:25:58.075116+0000 (oldest deadline 2024-01-24T14:26:27.475658+0000) 2024-01-24T14:26:28.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:28 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:28.391+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.647827+0000 front 2024-01-24T14:25:40.647938+0000 (oldest deadline 2024-01-24T14:26:05.347799+0000) 2024-01-24T14:26:28.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:28 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:28.391+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:45.348138+0000 front 2024-01-24T14:25:45.348156+0000 (oldest deadline 2024-01-24T14:26:08.848352+0000) 2024-01-24T14:26:28.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:28 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:28.391+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.749398+0000 front 2024-01-24T14:25:54.749444+0000 (oldest deadline 2024-01-24T14:26:18.849397+0000) 2024-01-24T14:26:28.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:28 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:28.391+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:25:58.849900+0000 front 2024-01-24T14:25:58.850006+0000 (oldest deadline 2024-01-24T14:26:24.749975+0000) 2024-01-24T14:26:28.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:28 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:28.481+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:28.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:28 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:28.481+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:28.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:28 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:28.481+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.424574+0000 front 2024-01-24T14:25:50.424437+0000 (oldest deadline 2024-01-24T14:26:17.425233+0000) 2024-01-24T14:26:29.485 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:29 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:29.226+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.092357+0000 front 2024-01-24T14:25:38.092298+0000 (oldest deadline 2024-01-24T14:26:03.992419+0000) 2024-01-24T14:26:29.486 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:29 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:29.226+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:43.993283+0000 front 2024-01-24T14:25:43.992896+0000 (oldest deadline 2024-01-24T14:26:09.892936+0000) 2024-01-24T14:26:29.486 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:29 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:29.226+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.993828+0000 front 2024-01-24T14:25:50.993806+0000 (oldest deadline 2024-01-24T14:26:17.394356+0000) 2024-01-24T14:26:29.486 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:29 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:29.226+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:26:02.496310+0000 front 2024-01-24T14:26:02.496373+0000 (oldest deadline 2024-01-24T14:26:27.096920+0000) 2024-01-24T14:26:29.524 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:29 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:29.143+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:39.060282+0000 front 2024-01-24T14:25:39.060365+0000 (oldest deadline 2024-01-24T14:26:02.560180+0000) 2024-01-24T14:26:29.524 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:29 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:29.143+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:47.261420+0000 front 2024-01-24T14:25:47.261437+0000 (oldest deadline 2024-01-24T14:26:10.161656+0000) 2024-01-24T14:26:29.524 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:29 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:29.143+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.763402+0000 front 2024-01-24T14:25:54.763474+0000 (oldest deadline 2024-01-24T14:26:19.463597+0000) 2024-01-24T14:26:29.524 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:29 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:29.143+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:26:01.164718+0000 front 2024-01-24T14:26:01.164658+0000 (oldest deadline 2024-01-24T14:26:25.264501+0000) 2024-01-24T14:26:29.799 INFO:journalctl@ceph.osd.7.smithi123.stdout:Jan 24 14:26:29 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-7[66344]: 2024-01-24T14:26:29.369+0000 7fd8d13e1700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:47.573271+0000 front 2024-01-24T14:25:47.573290+0000 (oldest deadline 2024-01-24T14:26:12.873464+0000) 2024-01-24T14:26:29.799 INFO:journalctl@ceph.osd.7.smithi123.stdout:Jan 24 14:26:29 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-7[66344]: 2024-01-24T14:26:29.369+0000 7fd8d13e1700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:55.174641+0000 front 2024-01-24T14:25:55.174696+0000 (oldest deadline 2024-01-24T14:26:18.074488+0000) 2024-01-24T14:26:29.800 INFO:journalctl@ceph.osd.7.smithi123.stdout:Jan 24 14:26:29 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-7[66344]: 2024-01-24T14:26:29.369+0000 7fd8d13e1700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:25:58.075039+0000 front 2024-01-24T14:25:58.075116+0000 (oldest deadline 2024-01-24T14:26:27.475658+0000) 2024-01-24T14:26:29.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:29 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:29.363+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.647827+0000 front 2024-01-24T14:25:40.647938+0000 (oldest deadline 2024-01-24T14:26:05.347799+0000) 2024-01-24T14:26:29.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:29 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:29.363+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:45.348138+0000 front 2024-01-24T14:25:45.348156+0000 (oldest deadline 2024-01-24T14:26:08.848352+0000) 2024-01-24T14:26:29.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:29 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:29.363+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.749398+0000 front 2024-01-24T14:25:54.749444+0000 (oldest deadline 2024-01-24T14:26:18.849397+0000) 2024-01-24T14:26:29.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:29 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:29.363+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:25:58.849900+0000 front 2024-01-24T14:25:58.850006+0000 (oldest deadline 2024-01-24T14:26:24.749975+0000) 2024-01-24T14:26:29.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:29 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:29.524+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:29.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:29 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:29.524+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:29.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:29 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:29.524+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.424574+0000 front 2024-01-24T14:25:50.424437+0000 (oldest deadline 2024-01-24T14:26:17.425233+0000) 2024-01-24T14:26:30.485 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:30 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:30.220+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.092357+0000 front 2024-01-24T14:25:38.092298+0000 (oldest deadline 2024-01-24T14:26:03.992419+0000) 2024-01-24T14:26:30.486 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:30 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:30.220+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:43.993283+0000 front 2024-01-24T14:25:43.992896+0000 (oldest deadline 2024-01-24T14:26:09.892936+0000) 2024-01-24T14:26:30.486 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:30 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:30.220+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.993828+0000 front 2024-01-24T14:25:50.993806+0000 (oldest deadline 2024-01-24T14:26:17.394356+0000) 2024-01-24T14:26:30.486 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:30 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:30.220+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:26:02.496310+0000 front 2024-01-24T14:26:02.496373+0000 (oldest deadline 2024-01-24T14:26:27.096920+0000) 2024-01-24T14:26:30.542 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:30 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:30.179+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:39.060282+0000 front 2024-01-24T14:25:39.060365+0000 (oldest deadline 2024-01-24T14:26:02.560180+0000) 2024-01-24T14:26:30.542 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:30 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:30.179+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:47.261420+0000 front 2024-01-24T14:25:47.261437+0000 (oldest deadline 2024-01-24T14:26:10.161656+0000) 2024-01-24T14:26:30.542 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:30 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:30.179+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.763402+0000 front 2024-01-24T14:25:54.763474+0000 (oldest deadline 2024-01-24T14:26:19.463597+0000) 2024-01-24T14:26:30.542 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:30 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:30.179+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:26:01.164718+0000 front 2024-01-24T14:26:01.164658+0000 (oldest deadline 2024-01-24T14:26:25.264501+0000) 2024-01-24T14:26:30.799 INFO:journalctl@ceph.osd.7.smithi123.stdout:Jan 24 14:26:30 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-7[66344]: 2024-01-24T14:26:30.355+0000 7fd8d13e1700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:47.573271+0000 front 2024-01-24T14:25:47.573290+0000 (oldest deadline 2024-01-24T14:26:12.873464+0000) 2024-01-24T14:26:30.799 INFO:journalctl@ceph.osd.7.smithi123.stdout:Jan 24 14:26:30 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-7[66344]: 2024-01-24T14:26:30.355+0000 7fd8d13e1700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:55.174641+0000 front 2024-01-24T14:25:55.174696+0000 (oldest deadline 2024-01-24T14:26:18.074488+0000) 2024-01-24T14:26:30.799 INFO:journalctl@ceph.osd.7.smithi123.stdout:Jan 24 14:26:30 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-7[66344]: 2024-01-24T14:26:30.355+0000 7fd8d13e1700 -1 osd.7 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:25:58.075039+0000 front 2024-01-24T14:25:58.075116+0000 (oldest deadline 2024-01-24T14:26:27.475658+0000) 2024-01-24T14:26:30.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:30 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:30.398+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.647827+0000 front 2024-01-24T14:25:40.647938+0000 (oldest deadline 2024-01-24T14:26:05.347799+0000) 2024-01-24T14:26:30.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:30 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:30.398+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:45.348138+0000 front 2024-01-24T14:25:45.348156+0000 (oldest deadline 2024-01-24T14:26:08.848352+0000) 2024-01-24T14:26:30.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:30 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:30.398+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.749398+0000 front 2024-01-24T14:25:54.749444+0000 (oldest deadline 2024-01-24T14:26:18.849397+0000) 2024-01-24T14:26:30.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:30 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:30.398+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:25:58.849900+0000 front 2024-01-24T14:25:58.850006+0000 (oldest deadline 2024-01-24T14:26:24.749975+0000) 2024-01-24T14:26:30.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:30 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:30.542+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:30.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:30 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:30.542+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:30.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:30 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:30.542+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.424574+0000 front 2024-01-24T14:25:50.424437+0000 (oldest deadline 2024-01-24T14:26:17.425233+0000) 2024-01-24T14:26:30.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:30 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:30.542+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:04.927419+0000 front 2024-01-24T14:26:04.927374+0000 (oldest deadline 2024-01-24T14:26:30.227348+0000) 2024-01-24T14:26:31.549 INFO:journalctl@ceph.osd.7.smithi123.stdout:Jan 24 14:26:31 smithi123 bash[83557]: ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-7 2024-01-24T14:26:31.550 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:31 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:31.360+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.647827+0000 front 2024-01-24T14:25:40.647938+0000 (oldest deadline 2024-01-24T14:26:05.347799+0000) 2024-01-24T14:26:31.550 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:31 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:31.360+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:45.348138+0000 front 2024-01-24T14:25:45.348156+0000 (oldest deadline 2024-01-24T14:26:08.848352+0000) 2024-01-24T14:26:31.550 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:31 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:31.360+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.749398+0000 front 2024-01-24T14:25:54.749444+0000 (oldest deadline 2024-01-24T14:26:18.849397+0000) 2024-01-24T14:26:31.550 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:31 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:31.360+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:25:58.849900+0000 front 2024-01-24T14:25:58.850006+0000 (oldest deadline 2024-01-24T14:26:24.749975+0000) 2024-01-24T14:26:31.550 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:31 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:31.360+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:08.850788+0000 front 2024-01-24T14:26:08.850811+0000 (oldest deadline 2024-01-24T14:26:30.550931+0000) 2024-01-24T14:26:31.555 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:31 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:31.135+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:39.060282+0000 front 2024-01-24T14:25:39.060365+0000 (oldest deadline 2024-01-24T14:26:02.560180+0000) 2024-01-24T14:26:31.555 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:31 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:31.135+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:47.261420+0000 front 2024-01-24T14:25:47.261437+0000 (oldest deadline 2024-01-24T14:26:10.161656+0000) 2024-01-24T14:26:31.556 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:31 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:31.135+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.763402+0000 front 2024-01-24T14:25:54.763474+0000 (oldest deadline 2024-01-24T14:26:19.463597+0000) 2024-01-24T14:26:31.556 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:31 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:31.135+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:26:01.164718+0000 front 2024-01-24T14:26:01.164658+0000 (oldest deadline 2024-01-24T14:26:25.264501+0000) 2024-01-24T14:26:31.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:31 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:31.245+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.092357+0000 front 2024-01-24T14:25:38.092298+0000 (oldest deadline 2024-01-24T14:26:03.992419+0000) 2024-01-24T14:26:31.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:31 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:31.245+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:43.993283+0000 front 2024-01-24T14:25:43.992896+0000 (oldest deadline 2024-01-24T14:26:09.892936+0000) 2024-01-24T14:26:31.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:31 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:31.245+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.993828+0000 front 2024-01-24T14:25:50.993806+0000 (oldest deadline 2024-01-24T14:26:17.394356+0000) 2024-01-24T14:26:31.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:31 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:31.245+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:26:02.496310+0000 front 2024-01-24T14:26:02.496373+0000 (oldest deadline 2024-01-24T14:26:27.096920+0000) 2024-01-24T14:26:31.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:31 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:31.555+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:31.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:31 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:31.555+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:31.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:31 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:31.555+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.424574+0000 front 2024-01-24T14:25:50.424437+0000 (oldest deadline 2024-01-24T14:26:17.425233+0000) 2024-01-24T14:26:31.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:31 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:31.555+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:04.927419+0000 front 2024-01-24T14:26:04.927374+0000 (oldest deadline 2024-01-24T14:26:30.227348+0000) 2024-01-24T14:26:32.531 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:32 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:32.182+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:39.060282+0000 front 2024-01-24T14:25:39.060365+0000 (oldest deadline 2024-01-24T14:26:02.560180+0000) 2024-01-24T14:26:32.531 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:32 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:32.182+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:47.261420+0000 front 2024-01-24T14:25:47.261437+0000 (oldest deadline 2024-01-24T14:26:10.161656+0000) 2024-01-24T14:26:32.531 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:32 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:32.182+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.763402+0000 front 2024-01-24T14:25:54.763474+0000 (oldest deadline 2024-01-24T14:26:19.463597+0000) 2024-01-24T14:26:32.531 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:32 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:32.182+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:26:01.164718+0000 front 2024-01-24T14:26:01.164658+0000 (oldest deadline 2024-01-24T14:26:25.264501+0000) 2024-01-24T14:26:32.536 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:32 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:32.345+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.647827+0000 front 2024-01-24T14:25:40.647938+0000 (oldest deadline 2024-01-24T14:26:05.347799+0000) 2024-01-24T14:26:32.536 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:32 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:32.345+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:45.348138+0000 front 2024-01-24T14:25:45.348156+0000 (oldest deadline 2024-01-24T14:26:08.848352+0000) 2024-01-24T14:26:32.536 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:32 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:32.345+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.749398+0000 front 2024-01-24T14:25:54.749444+0000 (oldest deadline 2024-01-24T14:26:18.849397+0000) 2024-01-24T14:26:32.537 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:32 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:32.345+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:25:58.849900+0000 front 2024-01-24T14:25:58.850006+0000 (oldest deadline 2024-01-24T14:26:24.749975+0000) 2024-01-24T14:26:32.537 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:32 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:32.345+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:08.850788+0000 front 2024-01-24T14:26:08.850811+0000 (oldest deadline 2024-01-24T14:26:30.550931+0000) 2024-01-24T14:26:32.544 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@osd.7.service' 2024-01-24T14:26:32.583 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T14:26:32.584 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-01-24T14:26:32.584 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-01-24T14:26:32.584 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@osd.8 2024-01-24T14:26:32.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:32 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:32.265+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.092357+0000 front 2024-01-24T14:25:38.092298+0000 (oldest deadline 2024-01-24T14:26:03.992419+0000) 2024-01-24T14:26:32.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:32 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:32.265+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:43.993283+0000 front 2024-01-24T14:25:43.992896+0000 (oldest deadline 2024-01-24T14:26:09.892936+0000) 2024-01-24T14:26:32.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:32 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:32.265+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.993828+0000 front 2024-01-24T14:25:50.993806+0000 (oldest deadline 2024-01-24T14:26:17.394356+0000) 2024-01-24T14:26:32.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:32 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:32.265+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:26:02.496310+0000 front 2024-01-24T14:26:02.496373+0000 (oldest deadline 2024-01-24T14:26:27.096920+0000) 2024-01-24T14:26:32.824 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:32 smithi194 systemd[1]: Stopping Ceph osd.8 for 654b0c8e-bac3-11ee-95b1-87774f69a715... 2024-01-24T14:26:32.824 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:32 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:32.530+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:32.824 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:32 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:32.530+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:32.824 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:32 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:32.530+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.424574+0000 front 2024-01-24T14:25:50.424437+0000 (oldest deadline 2024-01-24T14:26:17.425233+0000) 2024-01-24T14:26:32.825 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:32 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:32.530+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:04.927419+0000 front 2024-01-24T14:26:04.927374+0000 (oldest deadline 2024-01-24T14:26:30.227348+0000) 2024-01-24T14:26:33.094 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:32 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:32.823+0000 7fc9dadea700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T14:26:33.094 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:32 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:32.823+0000 7fc9dadea700 -1 osd.8 82 *** Got signal Terminated *** 2024-01-24T14:26:33.094 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:32 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:32.823+0000 7fc9dadea700 -1 osd.8 82 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-01-24T14:26:33.486 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:33 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:33.224+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.092357+0000 front 2024-01-24T14:25:38.092298+0000 (oldest deadline 2024-01-24T14:26:03.992419+0000) 2024-01-24T14:26:33.486 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:33 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:33.224+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:43.993283+0000 front 2024-01-24T14:25:43.992896+0000 (oldest deadline 2024-01-24T14:26:09.892936+0000) 2024-01-24T14:26:33.486 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:33 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:33.224+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.993828+0000 front 2024-01-24T14:25:50.993806+0000 (oldest deadline 2024-01-24T14:26:17.394356+0000) 2024-01-24T14:26:33.486 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:33 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:33.224+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:26:02.496310+0000 front 2024-01-24T14:26:02.496373+0000 (oldest deadline 2024-01-24T14:26:27.096920+0000) 2024-01-24T14:26:33.486 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:33 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:33.224+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:07.598010+0000 front 2024-01-24T14:26:07.598113+0000 (oldest deadline 2024-01-24T14:26:32.297843+0000) 2024-01-24T14:26:33.570 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:33 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:33.135+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:39.060282+0000 front 2024-01-24T14:25:39.060365+0000 (oldest deadline 2024-01-24T14:26:02.560180+0000) 2024-01-24T14:26:33.570 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:33 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:33.135+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:47.261420+0000 front 2024-01-24T14:25:47.261437+0000 (oldest deadline 2024-01-24T14:26:10.161656+0000) 2024-01-24T14:26:33.571 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:33 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:33.135+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.763402+0000 front 2024-01-24T14:25:54.763474+0000 (oldest deadline 2024-01-24T14:26:19.463597+0000) 2024-01-24T14:26:33.571 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:33 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:33.135+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:26:01.164718+0000 front 2024-01-24T14:26:01.164658+0000 (oldest deadline 2024-01-24T14:26:25.264501+0000) 2024-01-24T14:26:33.571 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:33 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:33.135+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:07.565750+0000 front 2024-01-24T14:26:07.565791+0000 (oldest deadline 2024-01-24T14:26:32.265721+0000) 2024-01-24T14:26:33.799 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:33 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:33.366+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.647827+0000 front 2024-01-24T14:25:40.647938+0000 (oldest deadline 2024-01-24T14:26:05.347799+0000) 2024-01-24T14:26:33.799 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:33 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:33.366+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:45.348138+0000 front 2024-01-24T14:25:45.348156+0000 (oldest deadline 2024-01-24T14:26:08.848352+0000) 2024-01-24T14:26:33.799 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:33 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:33.366+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.749398+0000 front 2024-01-24T14:25:54.749444+0000 (oldest deadline 2024-01-24T14:26:18.849397+0000) 2024-01-24T14:26:33.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:33 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:33.366+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:25:58.849900+0000 front 2024-01-24T14:25:58.850006+0000 (oldest deadline 2024-01-24T14:26:24.749975+0000) 2024-01-24T14:26:33.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:33 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:33.366+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:08.850788+0000 front 2024-01-24T14:26:08.850811+0000 (oldest deadline 2024-01-24T14:26:30.550931+0000) 2024-01-24T14:26:33.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:33 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:33.570+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:33.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:33 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:33.570+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:33.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:33 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:33.570+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.424574+0000 front 2024-01-24T14:25:50.424437+0000 (oldest deadline 2024-01-24T14:26:17.425233+0000) 2024-01-24T14:26:33.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:33 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:33.570+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:04.927419+0000 front 2024-01-24T14:26:04.927374+0000 (oldest deadline 2024-01-24T14:26:30.227348+0000) 2024-01-24T14:26:34.485 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:34 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:34.223+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.092357+0000 front 2024-01-24T14:25:38.092298+0000 (oldest deadline 2024-01-24T14:26:03.992419+0000) 2024-01-24T14:26:34.485 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:34 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:34.223+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:43.993283+0000 front 2024-01-24T14:25:43.992896+0000 (oldest deadline 2024-01-24T14:26:09.892936+0000) 2024-01-24T14:26:34.486 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:34 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:34.223+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.993828+0000 front 2024-01-24T14:25:50.993806+0000 (oldest deadline 2024-01-24T14:26:17.394356+0000) 2024-01-24T14:26:34.486 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:34 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:34.223+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:26:02.496310+0000 front 2024-01-24T14:26:02.496373+0000 (oldest deadline 2024-01-24T14:26:27.096920+0000) 2024-01-24T14:26:34.486 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:34 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:34.223+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:07.598010+0000 front 2024-01-24T14:26:07.598113+0000 (oldest deadline 2024-01-24T14:26:32.297843+0000) 2024-01-24T14:26:34.526 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:34 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:34.145+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:39.060282+0000 front 2024-01-24T14:25:39.060365+0000 (oldest deadline 2024-01-24T14:26:02.560180+0000) 2024-01-24T14:26:34.526 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:34 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:34.145+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:47.261420+0000 front 2024-01-24T14:25:47.261437+0000 (oldest deadline 2024-01-24T14:26:10.161656+0000) 2024-01-24T14:26:34.526 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:34 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:34.145+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.763402+0000 front 2024-01-24T14:25:54.763474+0000 (oldest deadline 2024-01-24T14:26:19.463597+0000) 2024-01-24T14:26:34.526 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:34 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:34.145+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:26:01.164718+0000 front 2024-01-24T14:26:01.164658+0000 (oldest deadline 2024-01-24T14:26:25.264501+0000) 2024-01-24T14:26:34.526 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:34 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:34.145+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:07.565750+0000 front 2024-01-24T14:26:07.565791+0000 (oldest deadline 2024-01-24T14:26:32.265721+0000) 2024-01-24T14:26:34.799 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:34 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:34.352+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.647827+0000 front 2024-01-24T14:25:40.647938+0000 (oldest deadline 2024-01-24T14:26:05.347799+0000) 2024-01-24T14:26:34.799 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:34 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:34.352+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:45.348138+0000 front 2024-01-24T14:25:45.348156+0000 (oldest deadline 2024-01-24T14:26:08.848352+0000) 2024-01-24T14:26:34.799 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:34 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:34.352+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.749398+0000 front 2024-01-24T14:25:54.749444+0000 (oldest deadline 2024-01-24T14:26:18.849397+0000) 2024-01-24T14:26:34.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:34 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:34.352+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:25:58.849900+0000 front 2024-01-24T14:25:58.850006+0000 (oldest deadline 2024-01-24T14:26:24.749975+0000) 2024-01-24T14:26:34.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:34 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:34.352+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:08.850788+0000 front 2024-01-24T14:26:08.850811+0000 (oldest deadline 2024-01-24T14:26:30.550931+0000) 2024-01-24T14:26:34.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:34 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:34.525+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:34.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:34 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:34.526+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:34.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:34 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:34.526+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.424574+0000 front 2024-01-24T14:25:50.424437+0000 (oldest deadline 2024-01-24T14:26:17.425233+0000) 2024-01-24T14:26:34.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:34 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:34.526+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:04.927419+0000 front 2024-01-24T14:26:04.927374+0000 (oldest deadline 2024-01-24T14:26:30.227348+0000) 2024-01-24T14:26:35.486 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:35 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:35.227+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.092357+0000 front 2024-01-24T14:25:38.092298+0000 (oldest deadline 2024-01-24T14:26:03.992419+0000) 2024-01-24T14:26:35.486 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:35 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:35.227+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:43.993283+0000 front 2024-01-24T14:25:43.992896+0000 (oldest deadline 2024-01-24T14:26:09.892936+0000) 2024-01-24T14:26:35.486 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:35 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:35.227+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.993828+0000 front 2024-01-24T14:25:50.993806+0000 (oldest deadline 2024-01-24T14:26:17.394356+0000) 2024-01-24T14:26:35.486 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:35 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:35.227+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:26:02.496310+0000 front 2024-01-24T14:26:02.496373+0000 (oldest deadline 2024-01-24T14:26:27.096920+0000) 2024-01-24T14:26:35.486 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:35 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:35.227+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:07.598010+0000 front 2024-01-24T14:26:07.598113+0000 (oldest deadline 2024-01-24T14:26:32.297843+0000) 2024-01-24T14:26:35.529 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:35 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:35.166+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:39.060282+0000 front 2024-01-24T14:25:39.060365+0000 (oldest deadline 2024-01-24T14:26:02.560180+0000) 2024-01-24T14:26:35.530 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:35 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:35.166+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:47.261420+0000 front 2024-01-24T14:25:47.261437+0000 (oldest deadline 2024-01-24T14:26:10.161656+0000) 2024-01-24T14:26:35.530 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:35 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:35.166+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.763402+0000 front 2024-01-24T14:25:54.763474+0000 (oldest deadline 2024-01-24T14:26:19.463597+0000) 2024-01-24T14:26:35.530 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:35 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:35.166+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:26:01.164718+0000 front 2024-01-24T14:26:01.164658+0000 (oldest deadline 2024-01-24T14:26:25.264501+0000) 2024-01-24T14:26:35.530 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:35 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:35.166+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:07.565750+0000 front 2024-01-24T14:26:07.565791+0000 (oldest deadline 2024-01-24T14:26:32.265721+0000) 2024-01-24T14:26:35.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:35 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:35.308+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.647827+0000 front 2024-01-24T14:25:40.647938+0000 (oldest deadline 2024-01-24T14:26:05.347799+0000) 2024-01-24T14:26:35.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:35 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:35.308+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:45.348138+0000 front 2024-01-24T14:25:45.348156+0000 (oldest deadline 2024-01-24T14:26:08.848352+0000) 2024-01-24T14:26:35.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:35 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:35.308+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.749398+0000 front 2024-01-24T14:25:54.749444+0000 (oldest deadline 2024-01-24T14:26:18.849397+0000) 2024-01-24T14:26:35.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:35 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:35.308+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:25:58.849900+0000 front 2024-01-24T14:25:58.850006+0000 (oldest deadline 2024-01-24T14:26:24.749975+0000) 2024-01-24T14:26:35.801 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:35 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:35.308+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:08.850788+0000 front 2024-01-24T14:26:08.850811+0000 (oldest deadline 2024-01-24T14:26:30.550931+0000) 2024-01-24T14:26:35.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:35 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:35.529+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:35.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:35 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:35.529+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:35.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:35 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:35.529+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.424574+0000 front 2024-01-24T14:25:50.424437+0000 (oldest deadline 2024-01-24T14:26:17.425233+0000) 2024-01-24T14:26:35.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:35 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:35.529+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:04.927419+0000 front 2024-01-24T14:26:04.927374+0000 (oldest deadline 2024-01-24T14:26:30.227348+0000) 2024-01-24T14:26:36.485 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:36 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:36.209+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.092357+0000 front 2024-01-24T14:25:38.092298+0000 (oldest deadline 2024-01-24T14:26:03.992419+0000) 2024-01-24T14:26:36.486 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:36 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:36.209+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:43.993283+0000 front 2024-01-24T14:25:43.992896+0000 (oldest deadline 2024-01-24T14:26:09.892936+0000) 2024-01-24T14:26:36.486 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:36 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:36.209+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.993828+0000 front 2024-01-24T14:25:50.993806+0000 (oldest deadline 2024-01-24T14:26:17.394356+0000) 2024-01-24T14:26:36.486 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:36 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:36.209+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:26:02.496310+0000 front 2024-01-24T14:26:02.496373+0000 (oldest deadline 2024-01-24T14:26:27.096920+0000) 2024-01-24T14:26:36.486 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:36 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:36.209+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:07.598010+0000 front 2024-01-24T14:26:07.598113+0000 (oldest deadline 2024-01-24T14:26:32.297843+0000) 2024-01-24T14:26:36.534 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:36 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:36.146+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:39.060282+0000 front 2024-01-24T14:25:39.060365+0000 (oldest deadline 2024-01-24T14:26:02.560180+0000) 2024-01-24T14:26:36.534 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:36 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:36.146+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:47.261420+0000 front 2024-01-24T14:25:47.261437+0000 (oldest deadline 2024-01-24T14:26:10.161656+0000) 2024-01-24T14:26:36.534 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:36 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:36.146+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.763402+0000 front 2024-01-24T14:25:54.763474+0000 (oldest deadline 2024-01-24T14:26:19.463597+0000) 2024-01-24T14:26:36.534 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:36 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:36.146+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:26:01.164718+0000 front 2024-01-24T14:26:01.164658+0000 (oldest deadline 2024-01-24T14:26:25.264501+0000) 2024-01-24T14:26:36.535 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:36 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:36.146+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:07.565750+0000 front 2024-01-24T14:26:07.565791+0000 (oldest deadline 2024-01-24T14:26:32.265721+0000) 2024-01-24T14:26:36.549 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:36 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:36.278+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.647827+0000 front 2024-01-24T14:25:40.647938+0000 (oldest deadline 2024-01-24T14:26:05.347799+0000) 2024-01-24T14:26:36.549 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:36 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:36.278+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:45.348138+0000 front 2024-01-24T14:25:45.348156+0000 (oldest deadline 2024-01-24T14:26:08.848352+0000) 2024-01-24T14:26:36.549 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:36 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:36.278+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.749398+0000 front 2024-01-24T14:25:54.749444+0000 (oldest deadline 2024-01-24T14:26:18.849397+0000) 2024-01-24T14:26:36.549 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:36 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:36.278+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:25:58.849900+0000 front 2024-01-24T14:25:58.850006+0000 (oldest deadline 2024-01-24T14:26:24.749975+0000) 2024-01-24T14:26:36.549 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:36 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:36.278+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:08.850788+0000 front 2024-01-24T14:26:08.850811+0000 (oldest deadline 2024-01-24T14:26:30.550931+0000) 2024-01-24T14:26:36.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:36 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:36.534+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:36.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:36 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:36.534+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:36.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:36 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:36.534+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.424574+0000 front 2024-01-24T14:25:50.424437+0000 (oldest deadline 2024-01-24T14:26:17.425233+0000) 2024-01-24T14:26:36.845 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:36 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:36.534+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:04.927419+0000 front 2024-01-24T14:26:04.927374+0000 (oldest deadline 2024-01-24T14:26:30.227348+0000) 2024-01-24T14:26:37.549 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:37 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:37.276+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.647827+0000 front 2024-01-24T14:25:40.647938+0000 (oldest deadline 2024-01-24T14:26:05.347799+0000) 2024-01-24T14:26:37.549 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:37 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:37.276+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:45.348138+0000 front 2024-01-24T14:25:45.348156+0000 (oldest deadline 2024-01-24T14:26:08.848352+0000) 2024-01-24T14:26:37.549 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:37 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:37.276+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.749398+0000 front 2024-01-24T14:25:54.749444+0000 (oldest deadline 2024-01-24T14:26:18.849397+0000) 2024-01-24T14:26:37.550 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:37 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:37.276+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:25:58.849900+0000 front 2024-01-24T14:25:58.850006+0000 (oldest deadline 2024-01-24T14:26:24.749975+0000) 2024-01-24T14:26:37.550 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:37 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:37.276+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:08.850788+0000 front 2024-01-24T14:26:08.850811+0000 (oldest deadline 2024-01-24T14:26:30.550931+0000) 2024-01-24T14:26:37.556 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:37 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:37.177+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:39.060282+0000 front 2024-01-24T14:25:39.060365+0000 (oldest deadline 2024-01-24T14:26:02.560180+0000) 2024-01-24T14:26:37.556 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:37 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:37.177+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:47.261420+0000 front 2024-01-24T14:25:47.261437+0000 (oldest deadline 2024-01-24T14:26:10.161656+0000) 2024-01-24T14:26:37.556 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:37 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:37.177+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.763402+0000 front 2024-01-24T14:25:54.763474+0000 (oldest deadline 2024-01-24T14:26:19.463597+0000) 2024-01-24T14:26:37.556 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:37 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:37.177+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:26:01.164718+0000 front 2024-01-24T14:26:01.164658+0000 (oldest deadline 2024-01-24T14:26:25.264501+0000) 2024-01-24T14:26:37.556 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:37 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8[67797]: 2024-01-24T14:26:37.177+0000 7fc9d4bdb700 -1 osd.8 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:07.565750+0000 front 2024-01-24T14:26:07.565791+0000 (oldest deadline 2024-01-24T14:26:32.265721+0000) 2024-01-24T14:26:37.735 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:37 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:37.257+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.092357+0000 front 2024-01-24T14:25:38.092298+0000 (oldest deadline 2024-01-24T14:26:03.992419+0000) 2024-01-24T14:26:37.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:37 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:37.257+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:43.993283+0000 front 2024-01-24T14:25:43.992896+0000 (oldest deadline 2024-01-24T14:26:09.892936+0000) 2024-01-24T14:26:37.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:37 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:37.257+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.993828+0000 front 2024-01-24T14:25:50.993806+0000 (oldest deadline 2024-01-24T14:26:17.394356+0000) 2024-01-24T14:26:37.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:37 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:37.257+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:26:02.496310+0000 front 2024-01-24T14:26:02.496373+0000 (oldest deadline 2024-01-24T14:26:27.096920+0000) 2024-01-24T14:26:37.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:37 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:37.257+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:07.598010+0000 front 2024-01-24T14:26:07.598113+0000 (oldest deadline 2024-01-24T14:26:32.297843+0000) 2024-01-24T14:26:37.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:37 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:37.556+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:37.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:37 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:37.556+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:37.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:37 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:37.556+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.424574+0000 front 2024-01-24T14:25:50.424437+0000 (oldest deadline 2024-01-24T14:26:17.425233+0000) 2024-01-24T14:26:37.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:37 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:37.556+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:04.927419+0000 front 2024-01-24T14:26:04.927374+0000 (oldest deadline 2024-01-24T14:26:30.227348+0000) 2024-01-24T14:26:38.549 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:38 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:38.266+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.647827+0000 front 2024-01-24T14:25:40.647938+0000 (oldest deadline 2024-01-24T14:26:05.347799+0000) 2024-01-24T14:26:38.549 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:38 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:38.266+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:45.348138+0000 front 2024-01-24T14:25:45.348156+0000 (oldest deadline 2024-01-24T14:26:08.848352+0000) 2024-01-24T14:26:38.549 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:38 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:38.266+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.749398+0000 front 2024-01-24T14:25:54.749444+0000 (oldest deadline 2024-01-24T14:26:18.849397+0000) 2024-01-24T14:26:38.549 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:38 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:38.266+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:25:58.849900+0000 front 2024-01-24T14:25:58.850006+0000 (oldest deadline 2024-01-24T14:26:24.749975+0000) 2024-01-24T14:26:38.549 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:38 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:38.266+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:08.850788+0000 front 2024-01-24T14:26:08.850811+0000 (oldest deadline 2024-01-24T14:26:30.550931+0000) 2024-01-24T14:26:38.735 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:38 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:38.271+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.092357+0000 front 2024-01-24T14:25:38.092298+0000 (oldest deadline 2024-01-24T14:26:03.992419+0000) 2024-01-24T14:26:38.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:38 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:38.271+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:43.993283+0000 front 2024-01-24T14:25:43.992896+0000 (oldest deadline 2024-01-24T14:26:09.892936+0000) 2024-01-24T14:26:38.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:38 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:38.271+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.993828+0000 front 2024-01-24T14:25:50.993806+0000 (oldest deadline 2024-01-24T14:26:17.394356+0000) 2024-01-24T14:26:38.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:38 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:38.271+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:26:02.496310+0000 front 2024-01-24T14:26:02.496373+0000 (oldest deadline 2024-01-24T14:26:27.096920+0000) 2024-01-24T14:26:38.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:38 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:38.271+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:07.598010+0000 front 2024-01-24T14:26:07.598113+0000 (oldest deadline 2024-01-24T14:26:32.297843+0000) 2024-01-24T14:26:38.844 INFO:journalctl@ceph.osd.8.smithi194.stdout:Jan 24 14:26:38 smithi194 bash[85280]: ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-8 2024-01-24T14:26:38.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:38 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:38.573+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:38.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:38 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:38.573+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:38.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:38 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:38.573+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.424574+0000 front 2024-01-24T14:25:50.424437+0000 (oldest deadline 2024-01-24T14:26:17.425233+0000) 2024-01-24T14:26:38.845 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:38 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:38.573+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:04.927419+0000 front 2024-01-24T14:26:04.927374+0000 (oldest deadline 2024-01-24T14:26:30.227348+0000) 2024-01-24T14:26:38.845 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:38 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:38.573+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-01-24T14:26:14.228558+0000 front 2024-01-24T14:26:14.228595+0000 (oldest deadline 2024-01-24T14:26:38.328651+0000) 2024-01-24T14:26:39.549 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:39 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:39.220+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.647827+0000 front 2024-01-24T14:25:40.647938+0000 (oldest deadline 2024-01-24T14:26:05.347799+0000) 2024-01-24T14:26:39.549 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:39 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:39.220+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:45.348138+0000 front 2024-01-24T14:25:45.348156+0000 (oldest deadline 2024-01-24T14:26:08.848352+0000) 2024-01-24T14:26:39.549 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:39 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:39.220+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.749398+0000 front 2024-01-24T14:25:54.749444+0000 (oldest deadline 2024-01-24T14:26:18.849397+0000) 2024-01-24T14:26:39.550 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:39 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:39.220+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:25:58.849900+0000 front 2024-01-24T14:25:58.850006+0000 (oldest deadline 2024-01-24T14:26:24.749975+0000) 2024-01-24T14:26:39.550 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:39 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:39.220+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:08.850788+0000 front 2024-01-24T14:26:08.850811+0000 (oldest deadline 2024-01-24T14:26:30.550931+0000) 2024-01-24T14:26:39.636 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@osd.8.service' 2024-01-24T14:26:39.681 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T14:26:39.681 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-01-24T14:26:39.681 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-01-24T14:26:39.681 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl stop ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@osd.9 2024-01-24T14:26:39.713 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:39 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:39.242+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.092357+0000 front 2024-01-24T14:25:38.092298+0000 (oldest deadline 2024-01-24T14:26:03.992419+0000) 2024-01-24T14:26:39.713 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:39 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:39.242+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:43.993283+0000 front 2024-01-24T14:25:43.992896+0000 (oldest deadline 2024-01-24T14:26:09.892936+0000) 2024-01-24T14:26:39.713 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:39 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:39.242+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.993828+0000 front 2024-01-24T14:25:50.993806+0000 (oldest deadline 2024-01-24T14:26:17.394356+0000) 2024-01-24T14:26:39.713 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:39 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:39.242+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:26:02.496310+0000 front 2024-01-24T14:26:02.496373+0000 (oldest deadline 2024-01-24T14:26:27.096920+0000) 2024-01-24T14:26:39.713 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:39 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:39.242+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:07.598010+0000 front 2024-01-24T14:26:07.598113+0000 (oldest deadline 2024-01-24T14:26:32.297843+0000) 2024-01-24T14:26:39.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:39 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:39.593+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:39.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:39 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:39.593+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:39.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:39 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:39.593+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.424574+0000 front 2024-01-24T14:25:50.424437+0000 (oldest deadline 2024-01-24T14:26:17.425233+0000) 2024-01-24T14:26:39.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:39 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:39.593+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:04.927419+0000 front 2024-01-24T14:26:04.927374+0000 (oldest deadline 2024-01-24T14:26:30.227348+0000) 2024-01-24T14:26:39.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:39 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:39.593+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-01-24T14:26:14.228558+0000 front 2024-01-24T14:26:14.228595+0000 (oldest deadline 2024-01-24T14:26:38.328651+0000) 2024-01-24T14:26:39.986 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:39 smithi019 systemd[1]: Stopping Ceph osd.9 for 654b0c8e-bac3-11ee-95b1-87774f69a715... 2024-01-24T14:26:39.986 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:39 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:39.942+0000 7f4c07ce5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T14:26:39.986 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:39 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:39.942+0000 7f4c07ce5700 -1 osd.9 82 *** Got signal Terminated *** 2024-01-24T14:26:39.986 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:39 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:39.942+0000 7f4c07ce5700 -1 osd.9 82 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-01-24T14:26:40.549 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:40 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:40.243+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.647827+0000 front 2024-01-24T14:25:40.647938+0000 (oldest deadline 2024-01-24T14:26:05.347799+0000) 2024-01-24T14:26:40.549 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:40 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:40.243+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:45.348138+0000 front 2024-01-24T14:25:45.348156+0000 (oldest deadline 2024-01-24T14:26:08.848352+0000) 2024-01-24T14:26:40.549 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:40 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:40.243+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.749398+0000 front 2024-01-24T14:25:54.749444+0000 (oldest deadline 2024-01-24T14:26:18.849397+0000) 2024-01-24T14:26:40.550 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:40 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:40.243+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:25:58.849900+0000 front 2024-01-24T14:25:58.850006+0000 (oldest deadline 2024-01-24T14:26:24.749975+0000) 2024-01-24T14:26:40.550 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:40 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:40.243+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:08.850788+0000 front 2024-01-24T14:26:08.850811+0000 (oldest deadline 2024-01-24T14:26:30.550931+0000) 2024-01-24T14:26:40.735 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:40 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:40.277+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.092357+0000 front 2024-01-24T14:25:38.092298+0000 (oldest deadline 2024-01-24T14:26:03.992419+0000) 2024-01-24T14:26:40.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:40 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:40.277+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:43.993283+0000 front 2024-01-24T14:25:43.992896+0000 (oldest deadline 2024-01-24T14:26:09.892936+0000) 2024-01-24T14:26:40.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:40 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:40.277+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.993828+0000 front 2024-01-24T14:25:50.993806+0000 (oldest deadline 2024-01-24T14:26:17.394356+0000) 2024-01-24T14:26:40.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:40 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:40.277+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:26:02.496310+0000 front 2024-01-24T14:26:02.496373+0000 (oldest deadline 2024-01-24T14:26:27.096920+0000) 2024-01-24T14:26:40.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:40 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:40.277+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:07.598010+0000 front 2024-01-24T14:26:07.598113+0000 (oldest deadline 2024-01-24T14:26:32.297843+0000) 2024-01-24T14:26:41.094 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:40 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:40.641+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:41.094 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:40 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:40.641+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:41.094 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:40 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:40.641+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.424574+0000 front 2024-01-24T14:25:50.424437+0000 (oldest deadline 2024-01-24T14:26:17.425233+0000) 2024-01-24T14:26:41.094 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:40 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:40.641+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:04.927419+0000 front 2024-01-24T14:26:04.927374+0000 (oldest deadline 2024-01-24T14:26:30.227348+0000) 2024-01-24T14:26:41.094 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:40 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:40.641+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-01-24T14:26:14.228558+0000 front 2024-01-24T14:26:14.228595+0000 (oldest deadline 2024-01-24T14:26:38.328651+0000) 2024-01-24T14:26:41.549 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:41 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:41.261+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.647827+0000 front 2024-01-24T14:25:40.647938+0000 (oldest deadline 2024-01-24T14:26:05.347799+0000) 2024-01-24T14:26:41.549 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:41 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:41.261+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:45.348138+0000 front 2024-01-24T14:25:45.348156+0000 (oldest deadline 2024-01-24T14:26:08.848352+0000) 2024-01-24T14:26:41.550 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:41 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:41.261+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.749398+0000 front 2024-01-24T14:25:54.749444+0000 (oldest deadline 2024-01-24T14:26:18.849397+0000) 2024-01-24T14:26:41.550 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:41 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:41.261+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:25:58.849900+0000 front 2024-01-24T14:25:58.850006+0000 (oldest deadline 2024-01-24T14:26:24.749975+0000) 2024-01-24T14:26:41.550 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:41 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:41.261+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:08.850788+0000 front 2024-01-24T14:26:08.850811+0000 (oldest deadline 2024-01-24T14:26:30.550931+0000) 2024-01-24T14:26:41.550 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:41 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:41.261+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-01-24T14:26:15.251791+0000 front 2024-01-24T14:26:15.251829+0000 (oldest deadline 2024-01-24T14:26:41.151781+0000) 2024-01-24T14:26:41.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:41 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:41.275+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.092357+0000 front 2024-01-24T14:25:38.092298+0000 (oldest deadline 2024-01-24T14:26:03.992419+0000) 2024-01-24T14:26:41.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:41 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:41.275+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:43.993283+0000 front 2024-01-24T14:25:43.992896+0000 (oldest deadline 2024-01-24T14:26:09.892936+0000) 2024-01-24T14:26:41.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:41 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:41.275+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.993828+0000 front 2024-01-24T14:25:50.993806+0000 (oldest deadline 2024-01-24T14:26:17.394356+0000) 2024-01-24T14:26:41.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:41 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:41.275+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:26:02.496310+0000 front 2024-01-24T14:26:02.496373+0000 (oldest deadline 2024-01-24T14:26:27.096920+0000) 2024-01-24T14:26:41.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:41 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:41.275+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:07.598010+0000 front 2024-01-24T14:26:07.598113+0000 (oldest deadline 2024-01-24T14:26:32.297843+0000) 2024-01-24T14:26:41.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:41 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:41.275+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-01-24T14:26:15.198940+0000 front 2024-01-24T14:26:15.198994+0000 (oldest deadline 2024-01-24T14:26:40.499011+0000) 2024-01-24T14:26:42.094 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:41 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:41.617+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:42.094 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:41 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:41.617+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:42.094 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:41 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:41.617+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.424574+0000 front 2024-01-24T14:25:50.424437+0000 (oldest deadline 2024-01-24T14:26:17.425233+0000) 2024-01-24T14:26:42.094 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:41 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:41.617+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:04.927419+0000 front 2024-01-24T14:26:04.927374+0000 (oldest deadline 2024-01-24T14:26:30.227348+0000) 2024-01-24T14:26:42.094 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:41 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:41.617+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-01-24T14:26:14.228558+0000 front 2024-01-24T14:26:14.228595+0000 (oldest deadline 2024-01-24T14:26:38.328651+0000) 2024-01-24T14:26:42.735 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:42 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:42.258+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.092357+0000 front 2024-01-24T14:25:38.092298+0000 (oldest deadline 2024-01-24T14:26:03.992419+0000) 2024-01-24T14:26:42.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:42 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:42.258+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:43.993283+0000 front 2024-01-24T14:25:43.992896+0000 (oldest deadline 2024-01-24T14:26:09.892936+0000) 2024-01-24T14:26:42.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:42 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:42.258+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.993828+0000 front 2024-01-24T14:25:50.993806+0000 (oldest deadline 2024-01-24T14:26:17.394356+0000) 2024-01-24T14:26:42.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:42 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:42.258+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:26:02.496310+0000 front 2024-01-24T14:26:02.496373+0000 (oldest deadline 2024-01-24T14:26:27.096920+0000) 2024-01-24T14:26:42.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:42 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:42.258+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:07.598010+0000 front 2024-01-24T14:26:07.598113+0000 (oldest deadline 2024-01-24T14:26:32.297843+0000) 2024-01-24T14:26:42.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:42 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:42.258+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-01-24T14:26:15.198940+0000 front 2024-01-24T14:26:15.198994+0000 (oldest deadline 2024-01-24T14:26:40.499011+0000) 2024-01-24T14:26:42.799 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:42 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:42.310+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.647827+0000 front 2024-01-24T14:25:40.647938+0000 (oldest deadline 2024-01-24T14:26:05.347799+0000) 2024-01-24T14:26:42.799 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:42 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:42.310+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:45.348138+0000 front 2024-01-24T14:25:45.348156+0000 (oldest deadline 2024-01-24T14:26:08.848352+0000) 2024-01-24T14:26:42.799 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:42 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:42.310+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.749398+0000 front 2024-01-24T14:25:54.749444+0000 (oldest deadline 2024-01-24T14:26:18.849397+0000) 2024-01-24T14:26:42.799 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:42 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:42.310+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:25:58.849900+0000 front 2024-01-24T14:25:58.850006+0000 (oldest deadline 2024-01-24T14:26:24.749975+0000) 2024-01-24T14:26:42.799 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:42 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:42.310+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:08.850788+0000 front 2024-01-24T14:26:08.850811+0000 (oldest deadline 2024-01-24T14:26:30.550931+0000) 2024-01-24T14:26:42.799 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:42 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:42.310+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-01-24T14:26:15.251791+0000 front 2024-01-24T14:26:15.251829+0000 (oldest deadline 2024-01-24T14:26:41.151781+0000) 2024-01-24T14:26:42.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:42 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:42.586+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:42.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:42 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:42.586+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:42.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:42 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:42.586+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.424574+0000 front 2024-01-24T14:25:50.424437+0000 (oldest deadline 2024-01-24T14:26:17.425233+0000) 2024-01-24T14:26:42.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:42 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:42.586+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:04.927419+0000 front 2024-01-24T14:26:04.927374+0000 (oldest deadline 2024-01-24T14:26:30.227348+0000) 2024-01-24T14:26:42.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:42 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:42.586+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-01-24T14:26:14.228558+0000 front 2024-01-24T14:26:14.228595+0000 (oldest deadline 2024-01-24T14:26:38.328651+0000) 2024-01-24T14:26:43.735 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:43 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:43.284+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.092357+0000 front 2024-01-24T14:25:38.092298+0000 (oldest deadline 2024-01-24T14:26:03.992419+0000) 2024-01-24T14:26:43.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:43 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:43.284+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:43.993283+0000 front 2024-01-24T14:25:43.992896+0000 (oldest deadline 2024-01-24T14:26:09.892936+0000) 2024-01-24T14:26:43.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:43 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:43.284+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.993828+0000 front 2024-01-24T14:25:50.993806+0000 (oldest deadline 2024-01-24T14:26:17.394356+0000) 2024-01-24T14:26:43.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:43 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:43.284+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:26:02.496310+0000 front 2024-01-24T14:26:02.496373+0000 (oldest deadline 2024-01-24T14:26:27.096920+0000) 2024-01-24T14:26:43.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:43 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:43.284+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:07.598010+0000 front 2024-01-24T14:26:07.598113+0000 (oldest deadline 2024-01-24T14:26:32.297843+0000) 2024-01-24T14:26:43.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:43 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:43.284+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-01-24T14:26:15.198940+0000 front 2024-01-24T14:26:15.198994+0000 (oldest deadline 2024-01-24T14:26:40.499011+0000) 2024-01-24T14:26:43.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:43 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:43.346+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.647827+0000 front 2024-01-24T14:25:40.647938+0000 (oldest deadline 2024-01-24T14:26:05.347799+0000) 2024-01-24T14:26:43.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:43 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:43.346+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:45.348138+0000 front 2024-01-24T14:25:45.348156+0000 (oldest deadline 2024-01-24T14:26:08.848352+0000) 2024-01-24T14:26:43.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:43 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:43.346+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.749398+0000 front 2024-01-24T14:25:54.749444+0000 (oldest deadline 2024-01-24T14:26:18.849397+0000) 2024-01-24T14:26:43.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:43 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:43.346+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:25:58.849900+0000 front 2024-01-24T14:25:58.850006+0000 (oldest deadline 2024-01-24T14:26:24.749975+0000) 2024-01-24T14:26:43.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:43 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:43.346+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:08.850788+0000 front 2024-01-24T14:26:08.850811+0000 (oldest deadline 2024-01-24T14:26:30.550931+0000) 2024-01-24T14:26:43.801 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:43 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:43.346+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-01-24T14:26:15.251791+0000 front 2024-01-24T14:26:15.251829+0000 (oldest deadline 2024-01-24T14:26:41.151781+0000) 2024-01-24T14:26:43.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:43 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:43.555+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:43.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:43 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:43.555+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:43.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:43 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:43.555+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.424574+0000 front 2024-01-24T14:25:50.424437+0000 (oldest deadline 2024-01-24T14:26:17.425233+0000) 2024-01-24T14:26:43.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:43 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:43.555+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:04.927419+0000 front 2024-01-24T14:26:04.927374+0000 (oldest deadline 2024-01-24T14:26:30.227348+0000) 2024-01-24T14:26:43.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:43 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:43.555+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-01-24T14:26:14.228558+0000 front 2024-01-24T14:26:14.228595+0000 (oldest deadline 2024-01-24T14:26:38.328651+0000) 2024-01-24T14:26:44.735 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:44 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:44.254+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.092357+0000 front 2024-01-24T14:25:38.092298+0000 (oldest deadline 2024-01-24T14:26:03.992419+0000) 2024-01-24T14:26:44.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:44 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:44.254+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:43.993283+0000 front 2024-01-24T14:25:43.992896+0000 (oldest deadline 2024-01-24T14:26:09.892936+0000) 2024-01-24T14:26:44.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:44 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:44.254+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.993828+0000 front 2024-01-24T14:25:50.993806+0000 (oldest deadline 2024-01-24T14:26:17.394356+0000) 2024-01-24T14:26:44.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:44 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:44.254+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:26:02.496310+0000 front 2024-01-24T14:26:02.496373+0000 (oldest deadline 2024-01-24T14:26:27.096920+0000) 2024-01-24T14:26:44.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:44 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:44.254+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:07.598010+0000 front 2024-01-24T14:26:07.598113+0000 (oldest deadline 2024-01-24T14:26:32.297843+0000) 2024-01-24T14:26:44.736 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:44 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9[84971]: 2024-01-24T14:26:44.254+0000 7f4c032d9700 -1 osd.9 82 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-01-24T14:26:15.198940+0000 front 2024-01-24T14:26:15.198994+0000 (oldest deadline 2024-01-24T14:26:40.499011+0000) 2024-01-24T14:26:44.799 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:44 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:44.390+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.647827+0000 front 2024-01-24T14:25:40.647938+0000 (oldest deadline 2024-01-24T14:26:05.347799+0000) 2024-01-24T14:26:44.799 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:44 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:44.390+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:45.348138+0000 front 2024-01-24T14:25:45.348156+0000 (oldest deadline 2024-01-24T14:26:08.848352+0000) 2024-01-24T14:26:44.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:44 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:44.390+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.749398+0000 front 2024-01-24T14:25:54.749444+0000 (oldest deadline 2024-01-24T14:26:18.849397+0000) 2024-01-24T14:26:44.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:44 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:44.390+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:25:58.849900+0000 front 2024-01-24T14:25:58.850006+0000 (oldest deadline 2024-01-24T14:26:24.749975+0000) 2024-01-24T14:26:44.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:44 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:44.390+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:08.850788+0000 front 2024-01-24T14:26:08.850811+0000 (oldest deadline 2024-01-24T14:26:30.550931+0000) 2024-01-24T14:26:44.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:44 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:44.391+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-01-24T14:26:15.251791+0000 front 2024-01-24T14:26:15.251829+0000 (oldest deadline 2024-01-24T14:26:41.151781+0000) 2024-01-24T14:26:44.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:44 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:44.547+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:44.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:44 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:44.547+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:44.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:44 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:44.547+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.424574+0000 front 2024-01-24T14:25:50.424437+0000 (oldest deadline 2024-01-24T14:26:17.425233+0000) 2024-01-24T14:26:44.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:44 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:44.547+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:04.927419+0000 front 2024-01-24T14:26:04.927374+0000 (oldest deadline 2024-01-24T14:26:30.227348+0000) 2024-01-24T14:26:44.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:44 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:44.547+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-01-24T14:26:14.228558+0000 front 2024-01-24T14:26:14.228595+0000 (oldest deadline 2024-01-24T14:26:38.328651+0000) 2024-01-24T14:26:44.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:44 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:44.547+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-01-24T14:26:18.329213+0000 front 2024-01-24T14:26:18.329146+0000 (oldest deadline 2024-01-24T14:26:44.229103+0000) 2024-01-24T14:26:45.735 INFO:journalctl@ceph.osd.9.smithi019.stdout:Jan 24 14:26:45 smithi019 bash[98490]: ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-9 2024-01-24T14:26:45.799 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:45 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:45.402+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.647827+0000 front 2024-01-24T14:25:40.647938+0000 (oldest deadline 2024-01-24T14:26:05.347799+0000) 2024-01-24T14:26:45.799 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:45 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:45.402+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:45.348138+0000 front 2024-01-24T14:25:45.348156+0000 (oldest deadline 2024-01-24T14:26:08.848352+0000) 2024-01-24T14:26:45.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:45 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:45.402+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.749398+0000 front 2024-01-24T14:25:54.749444+0000 (oldest deadline 2024-01-24T14:26:18.849397+0000) 2024-01-24T14:26:45.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:45 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:45.402+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:25:58.849900+0000 front 2024-01-24T14:25:58.850006+0000 (oldest deadline 2024-01-24T14:26:24.749975+0000) 2024-01-24T14:26:45.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:45 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:45.402+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:08.850788+0000 front 2024-01-24T14:26:08.850811+0000 (oldest deadline 2024-01-24T14:26:30.550931+0000) 2024-01-24T14:26:45.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:45 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:45.402+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-01-24T14:26:15.251791+0000 front 2024-01-24T14:26:15.251829+0000 (oldest deadline 2024-01-24T14:26:41.151781+0000) 2024-01-24T14:26:45.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:45 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:45.507+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:45.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:45 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:45.507+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:45.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:45 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:45.507+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.424574+0000 front 2024-01-24T14:25:50.424437+0000 (oldest deadline 2024-01-24T14:26:17.425233+0000) 2024-01-24T14:26:45.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:45 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:45.507+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:04.927419+0000 front 2024-01-24T14:26:04.927374+0000 (oldest deadline 2024-01-24T14:26:30.227348+0000) 2024-01-24T14:26:45.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:45 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:45.507+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-01-24T14:26:14.228558+0000 front 2024-01-24T14:26:14.228595+0000 (oldest deadline 2024-01-24T14:26:38.328651+0000) 2024-01-24T14:26:45.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:45 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:45.507+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-01-24T14:26:18.329213+0000 front 2024-01-24T14:26:18.329146+0000 (oldest deadline 2024-01-24T14:26:44.229103+0000) 2024-01-24T14:26:46.659 DEBUG:teuthology.orchestra.run.smithi019:> sudo pkill -f 'journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@osd.9.service' 2024-01-24T14:26:46.751 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T14:26:46.751 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-01-24T14:26:46.751 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-01-24T14:26:46.751 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@osd.10 2024-01-24T14:26:46.784 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:46 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:46.411+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.647827+0000 front 2024-01-24T14:25:40.647938+0000 (oldest deadline 2024-01-24T14:26:05.347799+0000) 2024-01-24T14:26:46.784 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:46 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:46.411+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:45.348138+0000 front 2024-01-24T14:25:45.348156+0000 (oldest deadline 2024-01-24T14:26:08.848352+0000) 2024-01-24T14:26:46.784 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:46 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:46.411+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.749398+0000 front 2024-01-24T14:25:54.749444+0000 (oldest deadline 2024-01-24T14:26:18.849397+0000) 2024-01-24T14:26:46.784 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:46 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:46.411+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:25:58.849900+0000 front 2024-01-24T14:25:58.850006+0000 (oldest deadline 2024-01-24T14:26:24.749975+0000) 2024-01-24T14:26:46.784 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:46 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:46.411+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:08.850788+0000 front 2024-01-24T14:26:08.850811+0000 (oldest deadline 2024-01-24T14:26:30.550931+0000) 2024-01-24T14:26:46.785 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:46 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:46.411+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-01-24T14:26:15.251791+0000 front 2024-01-24T14:26:15.251829+0000 (oldest deadline 2024-01-24T14:26:41.151781+0000) 2024-01-24T14:26:46.785 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:46 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:46.411+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-01-24T14:26:21.652990+0000 front 2024-01-24T14:26:21.652936+0000 (oldest deadline 2024-01-24T14:26:45.752810+0000) 2024-01-24T14:26:46.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:46 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:46.505+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:46.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:46 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:46.505+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:46.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:46 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:46.505+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.424574+0000 front 2024-01-24T14:25:50.424437+0000 (oldest deadline 2024-01-24T14:26:17.425233+0000) 2024-01-24T14:26:46.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:46 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:46.505+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:04.927419+0000 front 2024-01-24T14:26:04.927374+0000 (oldest deadline 2024-01-24T14:26:30.227348+0000) 2024-01-24T14:26:46.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:46 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:46.505+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-01-24T14:26:14.228558+0000 front 2024-01-24T14:26:14.228595+0000 (oldest deadline 2024-01-24T14:26:38.328651+0000) 2024-01-24T14:26:46.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:46 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:46.505+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-01-24T14:26:18.329213+0000 front 2024-01-24T14:26:18.329146+0000 (oldest deadline 2024-01-24T14:26:44.229103+0000) 2024-01-24T14:26:47.049 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:46 smithi123 systemd[1]: Stopping Ceph osd.10 for 654b0c8e-bac3-11ee-95b1-87774f69a715... 2024-01-24T14:26:47.049 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:47 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:47.003+0000 7f6ba1817700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T14:26:47.050 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:47 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:47.003+0000 7f6ba1817700 -1 osd.10 82 *** Got signal Terminated *** 2024-01-24T14:26:47.050 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:47 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:47.003+0000 7f6ba1817700 -1 osd.10 82 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-01-24T14:26:47.799 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:47 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:47.461+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.647827+0000 front 2024-01-24T14:25:40.647938+0000 (oldest deadline 2024-01-24T14:26:05.347799+0000) 2024-01-24T14:26:47.799 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:47 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:47.461+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:45.348138+0000 front 2024-01-24T14:25:45.348156+0000 (oldest deadline 2024-01-24T14:26:08.848352+0000) 2024-01-24T14:26:47.799 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:47 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:47.461+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.749398+0000 front 2024-01-24T14:25:54.749444+0000 (oldest deadline 2024-01-24T14:26:18.849397+0000) 2024-01-24T14:26:47.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:47 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:47.461+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:25:58.849900+0000 front 2024-01-24T14:25:58.850006+0000 (oldest deadline 2024-01-24T14:26:24.749975+0000) 2024-01-24T14:26:47.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:47 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:47.461+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:08.850788+0000 front 2024-01-24T14:26:08.850811+0000 (oldest deadline 2024-01-24T14:26:30.550931+0000) 2024-01-24T14:26:47.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:47 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:47.461+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-01-24T14:26:15.251791+0000 front 2024-01-24T14:26:15.251829+0000 (oldest deadline 2024-01-24T14:26:41.151781+0000) 2024-01-24T14:26:47.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:47 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:47.461+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-01-24T14:26:21.652990+0000 front 2024-01-24T14:26:21.652936+0000 (oldest deadline 2024-01-24T14:26:45.752810+0000) 2024-01-24T14:26:47.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:47 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:47.529+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:47.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:47 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:47.529+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:47.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:47 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:47.529+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.424574+0000 front 2024-01-24T14:25:50.424437+0000 (oldest deadline 2024-01-24T14:26:17.425233+0000) 2024-01-24T14:26:47.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:47 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:47.529+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:04.927419+0000 front 2024-01-24T14:26:04.927374+0000 (oldest deadline 2024-01-24T14:26:30.227348+0000) 2024-01-24T14:26:47.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:47 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:47.529+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-01-24T14:26:14.228558+0000 front 2024-01-24T14:26:14.228595+0000 (oldest deadline 2024-01-24T14:26:38.328651+0000) 2024-01-24T14:26:47.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:47 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:47.529+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-01-24T14:26:18.329213+0000 front 2024-01-24T14:26:18.329146+0000 (oldest deadline 2024-01-24T14:26:44.229103+0000) 2024-01-24T14:26:48.799 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:48 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:48.485+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.647827+0000 front 2024-01-24T14:25:40.647938+0000 (oldest deadline 2024-01-24T14:26:05.347799+0000) 2024-01-24T14:26:48.799 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:48 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:48.485+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:45.348138+0000 front 2024-01-24T14:25:45.348156+0000 (oldest deadline 2024-01-24T14:26:08.848352+0000) 2024-01-24T14:26:48.799 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:48 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:48.485+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.749398+0000 front 2024-01-24T14:25:54.749444+0000 (oldest deadline 2024-01-24T14:26:18.849397+0000) 2024-01-24T14:26:48.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:48 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:48.485+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:25:58.849900+0000 front 2024-01-24T14:25:58.850006+0000 (oldest deadline 2024-01-24T14:26:24.749975+0000) 2024-01-24T14:26:48.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:48 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:48.485+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:08.850788+0000 front 2024-01-24T14:26:08.850811+0000 (oldest deadline 2024-01-24T14:26:30.550931+0000) 2024-01-24T14:26:48.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:48 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:48.485+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-01-24T14:26:15.251791+0000 front 2024-01-24T14:26:15.251829+0000 (oldest deadline 2024-01-24T14:26:41.151781+0000) 2024-01-24T14:26:48.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:48 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:48.485+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-01-24T14:26:21.652990+0000 front 2024-01-24T14:26:21.652936+0000 (oldest deadline 2024-01-24T14:26:45.752810+0000) 2024-01-24T14:26:48.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:48 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:48.479+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:48.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:48 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:48.479+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:48.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:48 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:48.479+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.424574+0000 front 2024-01-24T14:25:50.424437+0000 (oldest deadline 2024-01-24T14:26:17.425233+0000) 2024-01-24T14:26:48.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:48 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:48.479+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:04.927419+0000 front 2024-01-24T14:26:04.927374+0000 (oldest deadline 2024-01-24T14:26:30.227348+0000) 2024-01-24T14:26:48.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:48 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:48.479+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-01-24T14:26:14.228558+0000 front 2024-01-24T14:26:14.228595+0000 (oldest deadline 2024-01-24T14:26:38.328651+0000) 2024-01-24T14:26:48.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:48 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:48.479+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-01-24T14:26:18.329213+0000 front 2024-01-24T14:26:18.329146+0000 (oldest deadline 2024-01-24T14:26:44.229103+0000) 2024-01-24T14:26:49.799 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:49 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:49.527+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.647827+0000 front 2024-01-24T14:25:40.647938+0000 (oldest deadline 2024-01-24T14:26:05.347799+0000) 2024-01-24T14:26:49.799 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:49 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:49.527+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:45.348138+0000 front 2024-01-24T14:25:45.348156+0000 (oldest deadline 2024-01-24T14:26:08.848352+0000) 2024-01-24T14:26:49.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:49 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:49.527+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.749398+0000 front 2024-01-24T14:25:54.749444+0000 (oldest deadline 2024-01-24T14:26:18.849397+0000) 2024-01-24T14:26:49.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:49 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:49.527+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:25:58.849900+0000 front 2024-01-24T14:25:58.850006+0000 (oldest deadline 2024-01-24T14:26:24.749975+0000) 2024-01-24T14:26:49.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:49 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:49.527+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:08.850788+0000 front 2024-01-24T14:26:08.850811+0000 (oldest deadline 2024-01-24T14:26:30.550931+0000) 2024-01-24T14:26:49.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:49 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:49.527+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-01-24T14:26:15.251791+0000 front 2024-01-24T14:26:15.251829+0000 (oldest deadline 2024-01-24T14:26:41.151781+0000) 2024-01-24T14:26:49.800 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:49 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:49.527+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-01-24T14:26:21.652990+0000 front 2024-01-24T14:26:21.652936+0000 (oldest deadline 2024-01-24T14:26:45.752810+0000) 2024-01-24T14:26:49.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:49 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:49.445+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:49.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:49 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:49.445+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:49.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:49 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:49.445+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.424574+0000 front 2024-01-24T14:25:50.424437+0000 (oldest deadline 2024-01-24T14:26:17.425233+0000) 2024-01-24T14:26:49.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:49 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:49.445+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:04.927419+0000 front 2024-01-24T14:26:04.927374+0000 (oldest deadline 2024-01-24T14:26:30.227348+0000) 2024-01-24T14:26:49.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:49 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:49.445+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-01-24T14:26:14.228558+0000 front 2024-01-24T14:26:14.228595+0000 (oldest deadline 2024-01-24T14:26:38.328651+0000) 2024-01-24T14:26:49.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:49 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:49.445+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-01-24T14:26:18.329213+0000 front 2024-01-24T14:26:18.329146+0000 (oldest deadline 2024-01-24T14:26:44.229103+0000) 2024-01-24T14:26:50.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:50 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:50.405+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:50.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:50 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:50.405+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:50.845 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:50 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:50.405+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.424574+0000 front 2024-01-24T14:25:50.424437+0000 (oldest deadline 2024-01-24T14:26:17.425233+0000) 2024-01-24T14:26:50.845 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:50 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:50.405+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:04.927419+0000 front 2024-01-24T14:26:04.927374+0000 (oldest deadline 2024-01-24T14:26:30.227348+0000) 2024-01-24T14:26:50.845 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:50 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:50.405+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-01-24T14:26:14.228558+0000 front 2024-01-24T14:26:14.228595+0000 (oldest deadline 2024-01-24T14:26:38.328651+0000) 2024-01-24T14:26:50.845 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:50 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:50.405+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-01-24T14:26:18.329213+0000 front 2024-01-24T14:26:18.329146+0000 (oldest deadline 2024-01-24T14:26:44.229103+0000) 2024-01-24T14:26:51.049 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:50 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:50.555+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.647827+0000 front 2024-01-24T14:25:40.647938+0000 (oldest deadline 2024-01-24T14:26:05.347799+0000) 2024-01-24T14:26:51.049 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:50 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:50.555+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:45.348138+0000 front 2024-01-24T14:25:45.348156+0000 (oldest deadline 2024-01-24T14:26:08.848352+0000) 2024-01-24T14:26:51.049 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:50 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:50.555+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.749398+0000 front 2024-01-24T14:25:54.749444+0000 (oldest deadline 2024-01-24T14:26:18.849397+0000) 2024-01-24T14:26:51.049 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:50 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:50.555+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:25:58.849900+0000 front 2024-01-24T14:25:58.850006+0000 (oldest deadline 2024-01-24T14:26:24.749975+0000) 2024-01-24T14:26:51.049 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:50 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:50.555+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:08.850788+0000 front 2024-01-24T14:26:08.850811+0000 (oldest deadline 2024-01-24T14:26:30.550931+0000) 2024-01-24T14:26:51.049 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:50 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:50.555+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-01-24T14:26:15.251791+0000 front 2024-01-24T14:26:15.251829+0000 (oldest deadline 2024-01-24T14:26:41.151781+0000) 2024-01-24T14:26:51.050 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:50 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:50.555+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-01-24T14:26:21.652990+0000 front 2024-01-24T14:26:21.652936+0000 (oldest deadline 2024-01-24T14:26:45.752810+0000) 2024-01-24T14:26:51.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:51 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:51.418+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:51.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:51 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:51.418+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:51.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:51 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:51.418+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.424574+0000 front 2024-01-24T14:25:50.424437+0000 (oldest deadline 2024-01-24T14:26:17.425233+0000) 2024-01-24T14:26:51.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:51 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:51.418+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:04.927419+0000 front 2024-01-24T14:26:04.927374+0000 (oldest deadline 2024-01-24T14:26:30.227348+0000) 2024-01-24T14:26:51.845 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:51 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:51.418+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-01-24T14:26:14.228558+0000 front 2024-01-24T14:26:14.228595+0000 (oldest deadline 2024-01-24T14:26:38.328651+0000) 2024-01-24T14:26:51.845 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:51 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:51.418+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-01-24T14:26:18.329213+0000 front 2024-01-24T14:26:18.329146+0000 (oldest deadline 2024-01-24T14:26:44.229103+0000) 2024-01-24T14:26:52.049 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:51 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:51.575+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:40.647827+0000 front 2024-01-24T14:25:40.647938+0000 (oldest deadline 2024-01-24T14:26:05.347799+0000) 2024-01-24T14:26:52.050 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:51 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:51.575+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:45.348138+0000 front 2024-01-24T14:25:45.348156+0000 (oldest deadline 2024-01-24T14:26:08.848352+0000) 2024-01-24T14:26:52.050 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:51 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:51.575+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:54.749398+0000 front 2024-01-24T14:25:54.749444+0000 (oldest deadline 2024-01-24T14:26:18.849397+0000) 2024-01-24T14:26:52.050 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:51 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:51.575+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6814 osd.3 since back 2024-01-24T14:25:58.849900+0000 front 2024-01-24T14:25:58.850006+0000 (oldest deadline 2024-01-24T14:26:24.749975+0000) 2024-01-24T14:26:52.050 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:51 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:51.575+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:08.850788+0000 front 2024-01-24T14:26:08.850811+0000 (oldest deadline 2024-01-24T14:26:30.550931+0000) 2024-01-24T14:26:52.050 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:51 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:51.575+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-01-24T14:26:15.251791+0000 front 2024-01-24T14:26:15.251829+0000 (oldest deadline 2024-01-24T14:26:41.151781+0000) 2024-01-24T14:26:52.050 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:51 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10[74940]: 2024-01-24T14:26:51.575+0000 7f6b9be09700 -1 osd.10 82 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-01-24T14:26:21.652990+0000 front 2024-01-24T14:26:21.652936+0000 (oldest deadline 2024-01-24T14:26:45.752810+0000) 2024-01-24T14:26:52.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:52 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:52.382+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:52.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:52 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:52.382+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:52.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:52 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:52.382+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.424574+0000 front 2024-01-24T14:25:50.424437+0000 (oldest deadline 2024-01-24T14:26:17.425233+0000) 2024-01-24T14:26:52.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:52 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:52.382+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:04.927419+0000 front 2024-01-24T14:26:04.927374+0000 (oldest deadline 2024-01-24T14:26:30.227348+0000) 2024-01-24T14:26:52.845 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:52 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:52.382+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-01-24T14:26:14.228558+0000 front 2024-01-24T14:26:14.228595+0000 (oldest deadline 2024-01-24T14:26:38.328651+0000) 2024-01-24T14:26:52.845 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:52 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:52.382+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-01-24T14:26:18.329213+0000 front 2024-01-24T14:26:18.329146+0000 (oldest deadline 2024-01-24T14:26:44.229103+0000) 2024-01-24T14:26:53.049 INFO:journalctl@ceph.osd.10.smithi123.stdout:Jan 24 14:26:52 smithi123 bash[83821]: ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-10 2024-01-24T14:26:53.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:53 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:53.393+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:53.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:53 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:53.393+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:53.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:53 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:53.393+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.424574+0000 front 2024-01-24T14:25:50.424437+0000 (oldest deadline 2024-01-24T14:26:17.425233+0000) 2024-01-24T14:26:53.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:53 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:53.393+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:04.927419+0000 front 2024-01-24T14:26:04.927374+0000 (oldest deadline 2024-01-24T14:26:30.227348+0000) 2024-01-24T14:26:53.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:53 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:53.393+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-01-24T14:26:14.228558+0000 front 2024-01-24T14:26:14.228595+0000 (oldest deadline 2024-01-24T14:26:38.328651+0000) 2024-01-24T14:26:53.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:53 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:53.393+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-01-24T14:26:18.329213+0000 front 2024-01-24T14:26:18.329146+0000 (oldest deadline 2024-01-24T14:26:44.229103+0000) 2024-01-24T14:26:53.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:53 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:53.393+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6820 osd.7 since back 2024-01-24T14:26:30.630486+0000 front 2024-01-24T14:26:30.630535+0000 (oldest deadline 2024-01-24T14:26:52.930415+0000) 2024-01-24T14:26:54.025 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@osd.10.service' 2024-01-24T14:26:54.106 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T14:26:54.106 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-01-24T14:26:54.106 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-01-24T14:26:54.106 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@osd.11 2024-01-24T14:26:54.438 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:54 smithi194 systemd[1]: Stopping Ceph osd.11 for 654b0c8e-bac3-11ee-95b1-87774f69a715... 2024-01-24T14:26:54.439 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:54 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:54.344+0000 7f2389e1d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T14:26:54.439 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:54 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:54.344+0000 7f2389e1d700 -1 osd.11 82 *** Got signal Terminated *** 2024-01-24T14:26:54.439 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:54 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:54.344+0000 7f2389e1d700 -1 osd.11 82 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-01-24T14:26:54.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:54 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:54.437+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:54.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:54 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:54.437+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:54.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:54 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:54.437+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.424574+0000 front 2024-01-24T14:25:50.424437+0000 (oldest deadline 2024-01-24T14:26:17.425233+0000) 2024-01-24T14:26:54.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:54 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:54.437+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:04.927419+0000 front 2024-01-24T14:26:04.927374+0000 (oldest deadline 2024-01-24T14:26:30.227348+0000) 2024-01-24T14:26:54.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:54 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:54.437+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-01-24T14:26:14.228558+0000 front 2024-01-24T14:26:14.228595+0000 (oldest deadline 2024-01-24T14:26:38.328651+0000) 2024-01-24T14:26:54.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:54 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:54.437+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-01-24T14:26:18.329213+0000 front 2024-01-24T14:26:18.329146+0000 (oldest deadline 2024-01-24T14:26:44.229103+0000) 2024-01-24T14:26:54.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:54 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:54.437+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6820 osd.7 since back 2024-01-24T14:26:30.630486+0000 front 2024-01-24T14:26:30.630535+0000 (oldest deadline 2024-01-24T14:26:52.930415+0000) 2024-01-24T14:26:55.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:55 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:55.463+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:55.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:55 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:55.463+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:55.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:55 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:55.463+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.424574+0000 front 2024-01-24T14:25:50.424437+0000 (oldest deadline 2024-01-24T14:26:17.425233+0000) 2024-01-24T14:26:55.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:55 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:55.463+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:04.927419+0000 front 2024-01-24T14:26:04.927374+0000 (oldest deadline 2024-01-24T14:26:30.227348+0000) 2024-01-24T14:26:55.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:55 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:55.463+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-01-24T14:26:14.228558+0000 front 2024-01-24T14:26:14.228595+0000 (oldest deadline 2024-01-24T14:26:38.328651+0000) 2024-01-24T14:26:55.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:55 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:55.463+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-01-24T14:26:18.329213+0000 front 2024-01-24T14:26:18.329146+0000 (oldest deadline 2024-01-24T14:26:44.229103+0000) 2024-01-24T14:26:55.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:55 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:55.463+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6820 osd.7 since back 2024-01-24T14:26:30.630486+0000 front 2024-01-24T14:26:30.630535+0000 (oldest deadline 2024-01-24T14:26:52.930415+0000) 2024-01-24T14:26:56.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:56 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:56.447+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:56.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:56 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:56.447+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:56.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:56 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:56.447+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.424574+0000 front 2024-01-24T14:25:50.424437+0000 (oldest deadline 2024-01-24T14:26:17.425233+0000) 2024-01-24T14:26:56.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:56 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:56.447+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:04.927419+0000 front 2024-01-24T14:26:04.927374+0000 (oldest deadline 2024-01-24T14:26:30.227348+0000) 2024-01-24T14:26:56.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:56 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:56.447+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-01-24T14:26:14.228558+0000 front 2024-01-24T14:26:14.228595+0000 (oldest deadline 2024-01-24T14:26:38.328651+0000) 2024-01-24T14:26:56.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:56 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:56.447+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-01-24T14:26:18.329213+0000 front 2024-01-24T14:26:18.329146+0000 (oldest deadline 2024-01-24T14:26:44.229103+0000) 2024-01-24T14:26:56.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:56 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:56.447+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6820 osd.7 since back 2024-01-24T14:26:30.630486+0000 front 2024-01-24T14:26:30.630535+0000 (oldest deadline 2024-01-24T14:26:52.930415+0000) 2024-01-24T14:26:57.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:57 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:57.472+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:57.847 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:57 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:57.472+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:57.847 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:57 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:57.472+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.424574+0000 front 2024-01-24T14:25:50.424437+0000 (oldest deadline 2024-01-24T14:26:17.425233+0000) 2024-01-24T14:26:57.847 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:57 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:57.472+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:04.927419+0000 front 2024-01-24T14:26:04.927374+0000 (oldest deadline 2024-01-24T14:26:30.227348+0000) 2024-01-24T14:26:57.847 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:57 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:57.472+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-01-24T14:26:14.228558+0000 front 2024-01-24T14:26:14.228595+0000 (oldest deadline 2024-01-24T14:26:38.328651+0000) 2024-01-24T14:26:57.847 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:57 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:57.472+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-01-24T14:26:18.329213+0000 front 2024-01-24T14:26:18.329146+0000 (oldest deadline 2024-01-24T14:26:44.229103+0000) 2024-01-24T14:26:57.847 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:57 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:57.472+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6820 osd.7 since back 2024-01-24T14:26:30.630486+0000 front 2024-01-24T14:26:30.630535+0000 (oldest deadline 2024-01-24T14:26:52.930415+0000) 2024-01-24T14:26:58.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:58 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:58.440+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6806 osd.0 since back 2024-01-24T14:25:38.722868+0000 front 2024-01-24T14:25:38.722885+0000 (oldest deadline 2024-01-24T14:26:02.223072+0000) 2024-01-24T14:26:58.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:58 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:58.440+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6804 osd.1 since back 2024-01-24T14:25:46.924251+0000 front 2024-01-24T14:25:46.924202+0000 (oldest deadline 2024-01-24T14:26:10.424135+0000) 2024-01-24T14:26:58.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:58 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:58.440+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-01-24T14:25:50.424574+0000 front 2024-01-24T14:25:50.424437+0000 (oldest deadline 2024-01-24T14:26:17.425233+0000) 2024-01-24T14:26:58.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:58 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:58.440+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6812 osd.4 since back 2024-01-24T14:26:04.927419+0000 front 2024-01-24T14:26:04.927374+0000 (oldest deadline 2024-01-24T14:26:30.227348+0000) 2024-01-24T14:26:58.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:58 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:58.440+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.194:6812 osd.5 since back 2024-01-24T14:26:14.228558+0000 front 2024-01-24T14:26:14.228595+0000 (oldest deadline 2024-01-24T14:26:38.328651+0000) 2024-01-24T14:26:58.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:58 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:58.440+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.19:6822 osd.6 since back 2024-01-24T14:26:18.329213+0000 front 2024-01-24T14:26:18.329146+0000 (oldest deadline 2024-01-24T14:26:44.229103+0000) 2024-01-24T14:26:58.844 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:58 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11[75905]: 2024-01-24T14:26:58.440+0000 7f2383c0e700 -1 osd.11 82 heartbeat_check: no reply from 172.21.15.123:6820 osd.7 since back 2024-01-24T14:26:30.630486+0000 front 2024-01-24T14:26:30.630535+0000 (oldest deadline 2024-01-24T14:26:52.930415+0000) 2024-01-24T14:27:00.344 INFO:journalctl@ceph.osd.11.smithi194.stdout:Jan 24 14:26:59 smithi194 bash[85543]: ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-osd-11 2024-01-24T14:27:01.091 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@osd.11.service' 2024-01-24T14:27:01.131 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T14:27:01.132 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-01-24T14:27:01.132 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-01-24T14:27:01.132 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl stop ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mds.a 2024-01-24T14:27:01.486 INFO:journalctl@ceph.mds.a.smithi019.stdout:Jan 24 14:27:01 smithi019 systemd[1]: Stopping Ceph mds.a for 654b0c8e-bac3-11ee-95b1-87774f69a715... 2024-01-24T14:27:01.486 INFO:journalctl@ceph.mds.a.smithi019.stdout:Jan 24 14:27:01 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-a[93471]: 2024-01-24T14:27:01.397+0000 7f68d1e0c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T14:27:01.486 INFO:journalctl@ceph.mds.a.smithi019.stdout:Jan 24 14:27:01 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-a[93471]: 2024-01-24T14:27:01.397+0000 7f68d1e0c700 -1 mds.a *** got signal Terminated *** 2024-01-24T14:27:04.101 INFO:journalctl@ceph.mds.a.smithi019.stdout:Jan 24 14:27:03 smithi019 bash[98749]: ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-a 2024-01-24T14:27:04.107 DEBUG:teuthology.orchestra.run.smithi019:> sudo pkill -f 'journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mds.a.service' 2024-01-24T14:27:04.149 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T14:27:04.149 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-01-24T14:27:04.150 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-01-24T14:27:04.150 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl stop ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mds.d 2024-01-24T14:27:04.430 INFO:journalctl@ceph.mds.d.smithi019.stdout:Jan 24 14:27:04 smithi019 systemd[1]: Stopping Ceph mds.d for 654b0c8e-bac3-11ee-95b1-87774f69a715... 2024-01-24T14:27:04.735 INFO:journalctl@ceph.mds.d.smithi019.stdout:Jan 24 14:27:04 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-d[92564]: 2024-01-24T14:27:04.428+0000 7fb57fbcb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T14:27:04.735 INFO:journalctl@ceph.mds.d.smithi019.stdout:Jan 24 14:27:04 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-d[92564]: 2024-01-24T14:27:04.428+0000 7fb57fbcb700 -1 mds.d *** got signal Terminated *** 2024-01-24T14:27:07.736 INFO:journalctl@ceph.mds.d.smithi019.stdout:Jan 24 14:27:07 smithi019 bash[98887]: ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-d 2024-01-24T14:27:07.955 DEBUG:teuthology.orchestra.run.smithi019:> sudo pkill -f 'journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mds.d.service' 2024-01-24T14:27:07.995 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T14:27:07.995 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-01-24T14:27:07.995 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-01-24T14:27:07.996 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl stop ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mds.g 2024-01-24T14:27:08.235 INFO:journalctl@ceph.mds.g.smithi019.stdout:Jan 24 14:27:08 smithi019 systemd[1]: Stopping Ceph mds.g for 654b0c8e-bac3-11ee-95b1-87774f69a715... 2024-01-24T14:27:08.735 INFO:journalctl@ceph.mds.g.smithi019.stdout:Jan 24 14:27:08 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-g[93084]: 2024-01-24T14:27:08.269+0000 7ff1d0900700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T14:27:08.736 INFO:journalctl@ceph.mds.g.smithi019.stdout:Jan 24 14:27:08 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-g[93084]: 2024-01-24T14:27:08.269+0000 7ff1d0900700 -1 mds.g *** got signal Terminated *** 2024-01-24T14:27:10.466 INFO:journalctl@ceph.mds.g.smithi019.stdout:Jan 24 14:27:10 smithi019 bash[99026]: ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-g 2024-01-24T14:27:10.472 DEBUG:teuthology.orchestra.run.smithi019:> sudo pkill -f 'journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mds.g.service' 2024-01-24T14:27:10.509 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T14:27:10.509 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-01-24T14:27:10.509 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-01-24T14:27:10.509 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl stop ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mds.j 2024-01-24T14:27:10.735 INFO:journalctl@ceph.mds.j.smithi019.stdout:Jan 24 14:27:10 smithi019 systemd[1]: Stopping Ceph mds.j for 654b0c8e-bac3-11ee-95b1-87774f69a715... 2024-01-24T14:27:11.236 INFO:journalctl@ceph.mds.j.smithi019.stdout:Jan 24 14:27:10 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-j[92177]: 2024-01-24T14:27:10.786+0000 7fd228c3c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T14:27:11.236 INFO:journalctl@ceph.mds.j.smithi019.stdout:Jan 24 14:27:10 smithi019 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-j[92177]: 2024-01-24T14:27:10.786+0000 7fd228c3c700 -1 mds.j *** got signal Terminated *** 2024-01-24T14:27:13.985 INFO:journalctl@ceph.mds.j.smithi019.stdout:Jan 24 14:27:13 smithi019 bash[99167]: ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-j 2024-01-24T14:27:14.014 DEBUG:teuthology.orchestra.run.smithi019:> sudo pkill -f 'journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mds.j.service' 2024-01-24T14:27:14.054 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T14:27:14.054 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-01-24T14:27:14.054 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-01-24T14:27:14.054 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mds.b 2024-01-24T14:27:14.549 INFO:journalctl@ceph.mds.b.smithi123.stdout:Jan 24 14:27:14 smithi123 systemd[1]: Stopping Ceph mds.b for 654b0c8e-bac3-11ee-95b1-87774f69a715... 2024-01-24T14:27:14.549 INFO:journalctl@ceph.mds.b.smithi123.stdout:Jan 24 14:27:14 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-b[81360]: 2024-01-24T14:27:14.307+0000 7f37fac9e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T14:27:14.550 INFO:journalctl@ceph.mds.b.smithi123.stdout:Jan 24 14:27:14 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-b[81360]: 2024-01-24T14:27:14.307+0000 7f37fac9e700 -1 mds.b *** got signal Terminated *** 2024-01-24T14:27:19.963 INFO:journalctl@ceph.mds.b.smithi123.stdout:Jan 24 14:27:19 smithi123 bash[84075]: ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-b 2024-01-24T14:27:19.969 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mds.b.service' 2024-01-24T14:27:20.005 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T14:27:20.006 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-01-24T14:27:20.006 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-01-24T14:27:20.006 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mds.e 2024-01-24T14:27:20.265 INFO:journalctl@ceph.mds.e.smithi123.stdout:Jan 24 14:27:20 smithi123 systemd[1]: Stopping Ceph mds.e for 654b0c8e-bac3-11ee-95b1-87774f69a715... 2024-01-24T14:27:20.549 INFO:journalctl@ceph.mds.e.smithi123.stdout:Jan 24 14:27:20 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-e[80677]: 2024-01-24T14:27:20.264+0000 7f3ebf8b1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T14:27:20.549 INFO:journalctl@ceph.mds.e.smithi123.stdout:Jan 24 14:27:20 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-e[80677]: 2024-01-24T14:27:20.264+0000 7f3ebf8b1700 -1 mds.e *** got signal Terminated *** 2024-01-24T14:27:23.243 INFO:journalctl@ceph.mds.e.smithi123.stdout:Jan 24 14:27:22 smithi123 bash[84215]: ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-e 2024-01-24T14:27:23.249 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mds.e.service' 2024-01-24T14:27:23.284 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T14:27:23.284 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-01-24T14:27:23.284 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-01-24T14:27:23.284 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mds.h 2024-01-24T14:27:23.547 INFO:journalctl@ceph.mds.h.smithi123.stdout:Jan 24 14:27:23 smithi123 systemd[1]: Stopping Ceph mds.h for 654b0c8e-bac3-11ee-95b1-87774f69a715... 2024-01-24T14:27:23.799 INFO:journalctl@ceph.mds.h.smithi123.stdout:Jan 24 14:27:23 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-h[81018]: 2024-01-24T14:27:23.547+0000 7feb9384a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T14:27:23.799 INFO:journalctl@ceph.mds.h.smithi123.stdout:Jan 24 14:27:23 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-h[81018]: 2024-01-24T14:27:23.547+0000 7feb9384a700 -1 mds.h *** got signal Terminated *** 2024-01-24T14:27:26.799 INFO:journalctl@ceph.mds.h.smithi123.stdout:Jan 24 14:27:26 smithi123 bash[84355]: ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-h 2024-01-24T14:27:26.865 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mds.h.service' 2024-01-24T14:27:26.898 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T14:27:26.899 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-01-24T14:27:26.899 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-01-24T14:27:26.899 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mds.k 2024-01-24T14:27:27.164 INFO:journalctl@ceph.mds.k.smithi123.stdout:Jan 24 14:27:26 smithi123 systemd[1]: Stopping Ceph mds.k for 654b0c8e-bac3-11ee-95b1-87774f69a715... 2024-01-24T14:27:27.549 INFO:journalctl@ceph.mds.k.smithi123.stdout:Jan 24 14:27:27 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-k[80324]: 2024-01-24T14:27:27.163+0000 7f3c44ff0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T14:27:27.550 INFO:journalctl@ceph.mds.k.smithi123.stdout:Jan 24 14:27:27 smithi123 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-k[80324]: 2024-01-24T14:27:27.163+0000 7f3c44ff0700 -1 mds.k *** got signal Terminated *** 2024-01-24T14:27:31.649 INFO:journalctl@ceph.mds.k.smithi123.stdout:Jan 24 14:27:31 smithi123 bash[84493]: ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-k 2024-01-24T14:27:31.655 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mds.k.service' 2024-01-24T14:27:31.688 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T14:27:31.688 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-01-24T14:27:31.688 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-01-24T14:27:31.688 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mds.c 2024-01-24T14:27:32.094 INFO:journalctl@ceph.mds.c.smithi194.stdout:Jan 24 14:27:31 smithi194 systemd[1]: Stopping Ceph mds.c for 654b0c8e-bac3-11ee-95b1-87774f69a715... 2024-01-24T14:27:32.094 INFO:journalctl@ceph.mds.c.smithi194.stdout:Jan 24 14:27:31 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-c[82746]: 2024-01-24T14:27:31.921+0000 7fbe62893700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T14:27:32.095 INFO:journalctl@ceph.mds.c.smithi194.stdout:Jan 24 14:27:31 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-c[82746]: 2024-01-24T14:27:31.921+0000 7fbe62893700 -1 mds.c *** got signal Terminated *** 2024-01-24T14:27:36.544 INFO:journalctl@ceph.mds.c.smithi194.stdout:Jan 24 14:27:36 smithi194 bash[85798]: ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-c 2024-01-24T14:27:36.550 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mds.c.service' 2024-01-24T14:27:36.589 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T14:27:36.590 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-01-24T14:27:36.590 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-01-24T14:27:36.590 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mds.f 2024-01-24T14:27:36.844 INFO:journalctl@ceph.mds.f.smithi194.stdout:Jan 24 14:27:36 smithi194 systemd[1]: Stopping Ceph mds.f for 654b0c8e-bac3-11ee-95b1-87774f69a715... 2024-01-24T14:27:37.344 INFO:journalctl@ceph.mds.f.smithi194.stdout:Jan 24 14:27:36 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-f[82406]: 2024-01-24T14:27:36.853+0000 7f0256d0d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T14:27:37.344 INFO:journalctl@ceph.mds.f.smithi194.stdout:Jan 24 14:27:36 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-f[82406]: 2024-01-24T14:27:36.853+0000 7f0256d0d700 -1 mds.f *** got signal Terminated *** 2024-01-24T14:27:43.207 INFO:journalctl@ceph.mds.f.smithi194.stdout:Jan 24 14:27:42 smithi194 bash[85944]: ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-f 2024-01-24T14:27:43.213 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mds.f.service' 2024-01-24T14:27:43.248 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T14:27:43.249 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-01-24T14:27:43.249 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-01-24T14:27:43.249 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mds.i 2024-01-24T14:27:43.512 INFO:journalctl@ceph.mds.i.smithi194.stdout:Jan 24 14:27:43 smithi194 systemd[1]: Stopping Ceph mds.i for 654b0c8e-bac3-11ee-95b1-87774f69a715... 2024-01-24T14:27:43.844 INFO:journalctl@ceph.mds.i.smithi194.stdout:Jan 24 14:27:43 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-i[83086]: 2024-01-24T14:27:43.511+0000 7f6b48362700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T14:27:43.844 INFO:journalctl@ceph.mds.i.smithi194.stdout:Jan 24 14:27:43 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-i[83086]: 2024-01-24T14:27:43.511+0000 7f6b48362700 -1 mds.i *** got signal Terminated *** 2024-01-24T14:27:46.039 INFO:journalctl@ceph.mds.i.smithi194.stdout:Jan 24 14:27:45 smithi194 bash[86083]: ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-i 2024-01-24T14:27:46.045 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mds.i.service' 2024-01-24T14:27:46.078 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T14:27:46.078 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-01-24T14:27:46.078 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-01-24T14:27:46.078 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mds.l 2024-01-24T14:27:46.336 INFO:journalctl@ceph.mds.l.smithi194.stdout:Jan 24 14:27:46 smithi194 systemd[1]: Stopping Ceph mds.l for 654b0c8e-bac3-11ee-95b1-87774f69a715... 2024-01-24T14:27:46.594 INFO:journalctl@ceph.mds.l.smithi194.stdout:Jan 24 14:27:46 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-l[82064]: 2024-01-24T14:27:46.335+0000 7fcf0182e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-01-24T14:27:46.594 INFO:journalctl@ceph.mds.l.smithi194.stdout:Jan 24 14:27:46 smithi194 ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-l[82064]: 2024-01-24T14:27:46.335+0000 7fcf0182e700 -1 mds.l *** got signal Terminated *** 2024-01-24T14:27:50.430 INFO:journalctl@ceph.mds.l.smithi194.stdout:Jan 24 14:27:50 smithi194 bash[86222]: ceph-654b0c8e-bac3-11ee-95b1-87774f69a715-mds-l 2024-01-24T14:27:50.436 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-654b0c8e-bac3-11ee-95b1-87774f69a715@mds.l.service' 2024-01-24T14:27:50.470 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-24T14:27:50.470 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-01-24T14:27:50.470 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 --force --keep-logs 2024-01-24T14:27:50.675 INFO:teuthology.orchestra.run.smithi019.stdout:Deleting cluster with fsid: 654b0c8e-bac3-11ee-95b1-87774f69a715 2024-01-24T14:27:52.235 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 --force --keep-logs 2024-01-24T14:27:52.426 INFO:teuthology.orchestra.run.smithi123.stdout:Deleting cluster with fsid: 654b0c8e-bac3-11ee-95b1-87774f69a715 2024-01-24T14:27:53.793 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 --force --keep-logs 2024-01-24T14:27:53.977 INFO:teuthology.orchestra.run.smithi194.stdout:Deleting cluster with fsid: 654b0c8e-bac3-11ee-95b1-87774f69a715 2024-01-24T14:27:55.366 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-01-24T14:27:55.404 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-01-24T14:27:55.441 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-01-24T14:27:55.471 INFO:tasks.cephadm:Archiving crash dumps... 2024-01-24T14:27:55.474 DEBUG:teuthology.misc:Transferring archived files from smithi019:/var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/crash to /home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530837/remote/smithi019/crash 2024-01-24T14:27:55.475 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar c -f - -C /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/crash -- . 2024-01-24T14:27:55.529 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/crash: Cannot open: No such file or directory 2024-01-24T14:27:55.530 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now 2024-01-24T14:27:55.532 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/crash to /home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530837/remote/smithi123/crash 2024-01-24T14:27:55.533 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/crash -- . 2024-01-24T14:27:55.585 INFO:teuthology.orchestra.run.smithi123.stderr:tar: /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/crash: Cannot open: No such file or directory 2024-01-24T14:27:55.585 INFO:teuthology.orchestra.run.smithi123.stderr:tar: Error is not recoverable: exiting now 2024-01-24T14:27:55.587 DEBUG:teuthology.misc:Transferring archived files from smithi194:/var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/crash to /home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530837/remote/smithi194/crash 2024-01-24T14:27:55.588 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/crash -- . 2024-01-24T14:27:55.687 INFO:teuthology.orchestra.run.smithi194.stderr:tar: /var/lib/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/crash: Cannot open: No such file or directory 2024-01-24T14:27:55.687 INFO:teuthology.orchestra.run.smithi194.stderr:tar: Error is not recoverable: exiting now 2024-01-24T14:27:55.688 INFO:tasks.cephadm:Checking cluster log for badness... 2024-01-24T14:27:55.688 DEBUG:teuthology.orchestra.run.smithi019:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-01-24T14:27:55.728 INFO:teuthology.orchestra.run.smithi019.stderr:grep: /var/log/ceph/654b0c8e-bac3-11ee-95b1-87774f69a715/ceph.log: No such file or directory 2024-01-24T14:27:55.730 INFO:tasks.cephadm:Compressing logs... 2024-01-24T14:27:55.730 DEBUG:teuthology.orchestra.run.smithi019:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-24T14:27:55.773 DEBUG:teuthology.orchestra.run.smithi123:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-24T14:27:55.776 DEBUG:teuthology.orchestra.run.smithi194:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-24T14:27:55.802 INFO:teuthology.orchestra.run.smithi194.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-01-24T14:27:55.806 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-01-24T14:27:55.807 INFO:teuthology.orchestra.run.smithi123.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-01-24T14:27:57.069 INFO:tasks.cephadm:Archiving logs... 2024-01-24T14:27:57.069 DEBUG:teuthology.misc:Transferring archived files from smithi019:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530837/remote/smithi019/log 2024-01-24T14:27:57.070 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-24T14:27:57.316 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530837/remote/smithi123/log 2024-01-24T14:27:57.317 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-24T14:27:57.467 DEBUG:teuthology.misc:Transferring archived files from smithi194:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530837/remote/smithi194/log 2024-01-24T14:27:57.468 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-24T14:27:57.614 INFO:tasks.cephadm:Removing cluster... 2024-01-24T14:27:57.614 DEBUG:teuthology.orchestra.run.smithi019:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 --force 2024-01-24T14:27:57.821 INFO:teuthology.orchestra.run.smithi019.stdout:Deleting cluster with fsid: 654b0c8e-bac3-11ee-95b1-87774f69a715 2024-01-24T14:27:58.083 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 --force 2024-01-24T14:27:58.269 INFO:teuthology.orchestra.run.smithi123.stdout:Deleting cluster with fsid: 654b0c8e-bac3-11ee-95b1-87774f69a715 2024-01-24T14:27:58.495 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 654b0c8e-bac3-11ee-95b1-87774f69a715 --force 2024-01-24T14:27:58.679 INFO:teuthology.orchestra.run.smithi194.stdout:Deleting cluster with fsid: 654b0c8e-bac3-11ee-95b1-87774f69a715 2024-01-24T14:27:58.909 INFO:tasks.cephadm:Removing cephadm ... 2024-01-24T14:27:58.909 DEBUG:teuthology.orchestra.run.smithi019:> rm -rf /home/ubuntu/cephtest/cephadm 2024-01-24T14:27:58.932 DEBUG:teuthology.orchestra.run.smithi123:> rm -rf /home/ubuntu/cephtest/cephadm 2024-01-24T14:27:58.954 DEBUG:teuthology.orchestra.run.smithi194:> rm -rf /home/ubuntu/cephtest/cephadm 2024-01-24T14:27:58.972 INFO:tasks.cephadm:Teardown complete 2024-01-24T14:27:58.973 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_4e3182a65800aed6d4aa2f6b86da3cfd72188134/qa/tasks/cephadm.py", line 1714, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_4e3182a65800aed6d4aa2f6b86da3cfd72188134/qa/tasks/cephadm.py", line 1629, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_4e3182a65800aed6d4aa2f6b86da3cfd72188134/qa/tasks/cephadm.py", line 1051, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/git.ceph.com_ceph-c_4e3182a65800aed6d4aa2f6b86da3cfd72188134/qa/tasks/cephfs/filesystem.py", line 532, in __init__ self.create() File "/home/teuthworker/src/git.ceph.com_ceph-c_4e3182a65800aed6d4aa2f6b86da3cfd72188134/qa/tasks/cephfs/filesystem.py", line 762, in create self.run_ceph_cmd('fs', 'subvolumegroup', 'create', self.name, 'qa') AttributeError: 'Filesystem' object has no attribute 'run_ceph_cmd' 2024-01-24T14:27:59.196 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=6ba684f636eb4729bccc62b3afff8696 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_4e3182a65800aed6d4aa2f6b86da3cfd72188134/qa/tasks/cephadm.py", line 1714, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_4e3182a65800aed6d4aa2f6b86da3cfd72188134/qa/tasks/cephadm.py", line 1629, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_4e3182a65800aed6d4aa2f6b86da3cfd72188134/qa/tasks/cephadm.py", line 1051, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/git.ceph.com_ceph-c_4e3182a65800aed6d4aa2f6b86da3cfd72188134/qa/tasks/cephfs/filesystem.py", line 532, in __init__ self.create() File "/home/teuthworker/src/git.ceph.com_ceph-c_4e3182a65800aed6d4aa2f6b86da3cfd72188134/qa/tasks/cephfs/filesystem.py", line 762, in create self.run_ceph_cmd('fs', 'subvolumegroup', 'create', self.name, 'qa') AttributeError: 'Filesystem' object has no attribute 'run_ceph_cmd' 2024-01-24T14:27:59.200 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-01-24T14:27:59.209 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-01-24T14:27:59.246 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-01-24T14:27:59.246 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-24T14:27:59.261 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-24T14:27:59.264 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-24T14:27:59.370 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-24T14:27:59.379 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-24T14:27:59.379 DEBUG:teuthology.orchestra.run.smithi019:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-24T14:27:59.382 DEBUG:teuthology.orchestra.run.smithi123:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-24T14:27:59.385 DEBUG:teuthology.orchestra.run.smithi194:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-24T14:27:59.400 INFO:teuthology.orchestra.run.smithi019.stderr:bash: ntpq: command not found 2024-01-24T14:27:59.400 INFO:teuthology.orchestra.run.smithi194.stderr:bash: ntpq: command not found 2024-01-24T14:27:59.402 INFO:teuthology.orchestra.run.smithi123.stderr:bash: ntpq: command not found 2024-01-24T14:27:59.406 INFO:teuthology.orchestra.run.smithi019.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-24T14:27:59.407 INFO:teuthology.orchestra.run.smithi019.stdout:=============================================================================== 2024-01-24T14:27:59.407 INFO:teuthology.orchestra.run.smithi019.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 29 -443us[ -402us] +/- 90ms 2024-01-24T14:27:59.407 INFO:teuthology.orchestra.run.smithi019.stdout:^* hv02.front.sepia.ceph.com 3 6 377 29 +70us[ +111us] +/- 52ms 2024-01-24T14:27:59.407 INFO:teuthology.orchestra.run.smithi019.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 28 +339us[ +339us] +/- 80ms 2024-01-24T14:27:59.407 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-01-24T14:27:59.407 INFO:teuthology.orchestra.run.smithi194.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-24T14:27:59.407 INFO:teuthology.orchestra.run.smithi194.stdout:=============================================================================== 2024-01-24T14:27:59.408 INFO:teuthology.orchestra.run.smithi194.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 28 -442us[ -442us] +/- 90ms 2024-01-24T14:27:59.408 INFO:teuthology.orchestra.run.smithi194.stdout:^* hv02.front.sepia.ceph.com 3 6 377 30 +68us[ +99us] +/- 52ms 2024-01-24T14:27:59.408 INFO:teuthology.orchestra.run.smithi194.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 30 +315us[ +346us] +/- 80ms 2024-01-24T14:27:59.408 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-01-24T14:27:59.409 INFO:teuthology.orchestra.run.smithi123.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-24T14:27:59.409 INFO:teuthology.orchestra.run.smithi123.stdout:=============================================================================== 2024-01-24T14:27:59.409 INFO:teuthology.orchestra.run.smithi123.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 29 -439us[ -405us] +/- 90ms 2024-01-24T14:27:59.409 INFO:teuthology.orchestra.run.smithi123.stdout:^* hv02.front.sepia.ceph.com 3 6 377 28 +73us[ +106us] +/- 52ms 2024-01-24T14:27:59.409 INFO:teuthology.orchestra.run.smithi123.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 30 +342us[ +376us] +/- 80ms 2024-01-24T14:27:59.410 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-01-24T14:27:59.410 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-24T14:27:59.419 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-24T14:27:59.420 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-24T14:27:59.437 DEBUG:teuthology.orchestra.run.smithi019:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-24T14:27:59.529 DEBUG:teuthology.orchestra.run.smithi123:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-24T14:27:59.601 DEBUG:teuthology.orchestra.run.smithi194:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-24T14:27:59.668 DEBUG:teuthology.orchestra.run.smithi019:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-24T14:27:59.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T14:27:59.707 DEBUG:teuthology.orchestra.run.smithi123:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-24T14:27:59.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T14:27:59.740 DEBUG:teuthology.orchestra.run.smithi194:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-24T14:27:59.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T14:27:59.770 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-24T14:27:59.779 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-24T14:27:59.813 INFO:teuthology.task.internal:Duration was 1370.054166 seconds 2024-01-24T14:27:59.814 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-24T14:27:59.847 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-24T14:27:59.847 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-24T14:27:59.850 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-24T14:27:59.852 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-24T14:27:59.899 INFO:teuthology.orchestra.run.smithi194.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-24T14:27:59.900 INFO:teuthology.orchestra.run.smithi123.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-24T14:27:59.907 INFO:teuthology.orchestra.run.smithi019.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-24T14:28:00.357 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-01-24T14:28:00.357 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi019.front.sepia.ceph.com 2024-01-24T14:28:00.357 DEBUG:teuthology.orchestra.run.smithi019:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-24T14:28:00.386 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi123.front.sepia.ceph.com 2024-01-24T14:28:00.386 DEBUG:teuthology.orchestra.run.smithi123:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-24T14:28:00.413 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi194.front.sepia.ceph.com 2024-01-24T14:28:00.414 DEBUG:teuthology.orchestra.run.smithi194:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-24T14:28:00.437 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-01-24T14:28:00.438 DEBUG:teuthology.orchestra.run.smithi019:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-24T14:28:00.440 DEBUG:teuthology.orchestra.run.smithi123:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-24T14:28:00.456 DEBUG:teuthology.orchestra.run.smithi194:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-24T14:28:00.529 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-01-24T14:28:00.529 DEBUG:teuthology.orchestra.run.smithi019:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-24T14:28:00.531 DEBUG:teuthology.orchestra.run.smithi123:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-24T14:28:00.534 DEBUG:teuthology.orchestra.run.smithi194:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-24T14:28:00.636 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-24T14:28:00.645 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-24T14:28:00.646 DEBUG:teuthology.orchestra.run.smithi019:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-24T14:28:00.677 DEBUG:teuthology.orchestra.run.smithi123:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-24T14:28:00.705 DEBUG:teuthology.orchestra.run.smithi194:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-24T14:28:00.732 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-24T14:28:00.767 DEBUG:teuthology.orchestra.run.smithi019:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-24T14:28:00.770 DEBUG:teuthology.orchestra.run.smithi123:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-24T14:28:00.772 DEBUG:teuthology.orchestra.run.smithi194:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-24T14:28:00.797 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern = core 2024-01-24T14:28:00.798 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern = core 2024-01-24T14:28:00.801 INFO:teuthology.orchestra.run.smithi019.stdout:kernel.core_pattern = core 2024-01-24T14:28:00.859 DEBUG:teuthology.orchestra.run.smithi019:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-24T14:28:00.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T14:28:00.880 DEBUG:teuthology.orchestra.run.smithi123:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-24T14:28:00.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T14:28:00.897 DEBUG:teuthology.orchestra.run.smithi194:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-24T14:28:00.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-24T14:28:00.916 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-24T14:28:00.926 INFO:teuthology.task.internal:Transferring archived files... 2024-01-24T14:28:00.927 DEBUG:teuthology.misc:Transferring archived files from smithi019:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530837/remote/smithi019 2024-01-24T14:28:00.927 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-24T14:28:00.977 DEBUG:teuthology.misc:Transferring archived files from smithi123:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530837/remote/smithi123 2024-01-24T14:28:00.978 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-24T14:28:01.017 DEBUG:teuthology.misc:Transferring archived files from smithi194:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530837/remote/smithi194 2024-01-24T14:28:01.017 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-24T14:28:01.057 INFO:teuthology.task.internal:Removing archive directory... 2024-01-24T14:28:01.057 DEBUG:teuthology.orchestra.run.smithi019:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-24T14:28:01.060 DEBUG:teuthology.orchestra.run.smithi123:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-24T14:28:01.062 DEBUG:teuthology.orchestra.run.smithi194:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-24T14:28:01.108 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-24T14:28:01.122 INFO:teuthology.task.internal:Not uploading archives. 2024-01-24T14:28:01.122 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-24T14:28:01.139 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-24T14:28:01.139 DEBUG:teuthology.orchestra.run.smithi019:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-24T14:28:01.142 DEBUG:teuthology.orchestra.run.smithi123:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-24T14:28:01.144 DEBUG:teuthology.orchestra.run.smithi194:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-24T14:28:01.158 INFO:teuthology.orchestra.run.smithi123.stdout: 262157 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 24 14:28 /home/ubuntu/cephtest 2024-01-24T14:28:01.161 INFO:teuthology.orchestra.run.smithi019.stdout: 262175 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 24 14:28 /home/ubuntu/cephtest 2024-01-24T14:28:01.164 INFO:teuthology.orchestra.run.smithi194.stdout: 262164 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 24 14:28 /home/ubuntu/cephtest 2024-01-24T14:28:01.166 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-24T14:28:01.177 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-24T14:28:01.239 INFO:teuthology.nuke:Checking targets against current locks 2024-01-24T14:28:01.263 DEBUG:teuthology.nuke:shortname: smithi019 2024-01-24T14:28:01.263 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-24T14:28:01.287 DEBUG:teuthology.nuke:shortname: smithi123 2024-01-24T14:28:01.287 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-24T14:28:01.292 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi019.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530837', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-24 13:59:42.252661', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHCj2vo9mBN4fsm+73SKJZM3YxS2M2g3IWt5gbUm6jW2ldYtekXzlf1XSmHQJrb7pQ24ra9oIpY2GFVq0Gz8r6M='} 2024-01-24T14:28:01.315 DEBUG:teuthology.nuke:shortname: smithi194 2024-01-24T14:28:01.315 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-24T14:28:01.320 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi123.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530837', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-24 13:59:42.254693', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBlD4CfikN1KNTYaVpAXMZ3p6qUfCC9hNLeMCEWVmFr6xcOslMZQhCDVE1zB4pZelVZoRtaXFpPsTqfx/brODcA='} 2024-01-24T14:28:01.351 INFO:teuthology.orchestra.console.smithi019:Power off 2024-01-24T14:28:01.351 DEBUG:teuthology.orchestra.console.smithi019:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-24T14:28:01.368 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi194.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-01-24_12:12:30-fs-wip-vshankar-testing1-reef-2024-01-24-1150-testing-default-smithi/7530837', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-24 13:59:42.256410', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAolgJUQEL2wZGHALo6/eF3ZJsFX6UwtYj4ZOuFPcXK8EvqTQ6Bc+sopfQrEDTmDpL1/QTlLkV7BOHONov0zftE='} 2024-01-24T14:28:01.380 DEBUG:teuthology.orchestra.console.smithi019:power off output: Chassis Power Control: Down/Off 2024-01-24T14:28:01.380 DEBUG:teuthology.orchestra.console.smithi019:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-24T14:28:01.400 INFO:teuthology.orchestra.console.smithi123:Power off 2024-01-24T14:28:01.401 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-24T14:28:01.414 DEBUG:teuthology.orchestra.console.smithi019:check power output: Chassis Power is on 2024-01-24T14:28:01.424 DEBUG:teuthology.orchestra.console.smithi123:power off output: Chassis Power Control: Down/Off 2024-01-24T14:28:01.424 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-24T14:28:01.442 INFO:teuthology.orchestra.console.smithi194:Power off 2024-01-24T14:28:01.442 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-24T14:28:01.452 DEBUG:teuthology.orchestra.console.smithi123:check power output: Chassis Power is on 2024-01-24T14:28:01.465 DEBUG:teuthology.orchestra.console.smithi194:power off output: Chassis Power Control: Down/Off 2024-01-24T14:28:01.465 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-24T14:28:01.489 DEBUG:teuthology.orchestra.console.smithi194:check power output: Chassis Power is on 2024-01-24T14:28:05.415 DEBUG:teuthology.orchestra.console.smithi019:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-24T14:28:05.453 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-24T14:28:05.490 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-24T14:28:05.527 DEBUG:teuthology.orchestra.console.smithi019:check power output: Chassis Power is on 2024-01-24T14:28:05.563 DEBUG:teuthology.orchestra.console.smithi123:check power output: Chassis Power is on 2024-01-24T14:28:05.601 DEBUG:teuthology.orchestra.console.smithi194:check power output: Chassis Power is on 2024-01-24T14:28:09.530 DEBUG:teuthology.orchestra.console.smithi019:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-24T14:28:09.564 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-24T14:28:09.602 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-24T14:28:09.644 DEBUG:teuthology.orchestra.console.smithi019:check power output: Chassis Power is off 2024-01-24T14:28:09.674 DEBUG:teuthology.orchestra.console.smithi123:check power output: Chassis Power is off 2024-01-24T14:28:09.712 DEBUG:teuthology.orchestra.console.smithi194:check power output: Chassis Power is on 2024-01-24T14:28:09.746 INFO:teuthology.orchestra.console.smithi019:Power off completed 2024-01-24T14:28:09.775 INFO:teuthology.orchestra.console.smithi123:Power off completed 2024-01-24T14:28:13.716 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-24T14:28:13.926 DEBUG:teuthology.orchestra.console.smithi194:check power output: Chassis Power is off 2024-01-24T14:28:14.026 INFO:teuthology.orchestra.console.smithi194:Power off completed 2024-01-24T14:28:14.132 INFO:teuthology.run:Summary data: description: fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/kernel_untar_build}} duration: 1370.0541656017303 failure_reason: '''Filesystem'' object has no attribute ''run_ceph_cmd''' flavor: default owner: scheduled_vshankar@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=6ba684f636eb4729bccc62b3afff8696 status: fail success: false 2024-01-24T14:28:14.133 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-24T14:28:14.215 INFO:teuthology.run:FAIL